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I. Introduction 

Research into representations of Lie groups was motivated on the one hand by 
physics, and on the other hand by the theory of automorphic forms. The theory 
of unitary or admissible representations of noncompact reductive Lie groups has 
been developed systematically and intensively shortly after the end of World War 

II. In particular, Harish-Chandra, R. Langlands, Gelfand school and some other 
people made an enormous contribution to the theory of unitary representations of 
noncompact reductive Lie groups. 

Early in the 1960s A. A. Kirillov 07J first initiated the orbit method for a nilpo- 
tent real Lie group attaching an irreducible unitary representation to a coadjoint 
orbit (which is a homogeneous symplectic manifold) in a perfect way. Thereafter 
Kirillov's work was generalized to solvable groups of type I by L. Auslander and B. 
Kostant [3] early in the 1970s in a nice way. Their proof was based on the existence 
of complex polarizations satisfying a positivity condition. Unfortunately Kirillov's 
work fails to be generalized in some ways to the case of compact Lie groups or 
semisimple Lie groups. Relatively simple groups like «SX(2, M) have irreducible uni- 
tary representations that do not correspond to any symplectic homogeneous space. 
Conversely, P. Torasso [85] found that the double cover of SL(3, R) has a homoge- 
neous symplectic manifold corresponding to no unitary representations. The orbit 
method for reductive Lie groups is a kind of a philosophy but not a theorem. Many 
large families of orbits correspond in comprehensible ways to unitary representa- 
tions, and provide a clear geometric picture of these representations. The coadjoint 
orbits for a reductive Lie group are classified into three kinds of orbits, namely, 
hyperbolic, elliptic and nilpotent ones. The hyperbolic orbits are related to the uni- 
tary representations obtained by the parabolic induction and on the other hand, the 
elliptic ones are related to the unitary representations obtained by the cohomolog- 
ical induction. However, we still have no idea of attaching unitary representations 
to nilpotent orbits. It is known that there are only finitely many nilpotent orbits. 
In a certain case, some nilpotent orbits are corresponded to the so-called unipotent 
representations. For instance, a minimal nilpotent orbit is attached to a minimal 
representation. In fact, the notion of unipotent representations is not still well de- 
fined. The investigation of unipotent representations is now under way. Recently 
D. Vogan 93 presented a new method for studying the quantization of nilpotent 
orbits in terms of the restriction to a maximal compact subgroup even though it is 
not complete and is in a preliminary stage. J.-S. Huang and J.-S. Li attached 
unitary representations to spherical nilpotent orbits for the real orthogonal and 
symplectic groups. 
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In this article, we describe a development of the orbit method for real Lie groups, 
and then in particular, we study the orbit method for the Heisenberg group and the 
Jacobi group in detail. This paper is organized as follows. In Section 2, we describe 
the notion of geometric quantization relating to the theory of unitary representa- 
tions which led to the orbit method. The study of the geometric quantization was 
first made intensively by A. A. Kirillov [5T] . In Section 3, we outline the beautiful 
Kirillov's work on the orbit method for a nilpotent real Lie group done early in 
the 1960s. In Section 4, we describe the work for a solvable Lie group of type I 
done by L. Auslander and B. Kostant |3j generalizing Kirillov's work. In Section 5, 
we roughly discuss the cases of compact or semisimple Lie groups where the orbit 
method does not work nicely. If G is compact or semisimple, the correspondence 
between G-orbits and irreducible unitary representations of G breaks down. In 
Section 6, for a real reductive Lie group G with Lie algebra g, we present some 
properties of nilpotent orbits for G and describe the Kostant-Sekiguchi correspon- 
dence between G-orbits in the cone of all nilpotent elements in g and i^c-orbits in 
the cone of nilpotent elements on pc, where Kc is the complexification of a fixed 
maximal compact subgroup K of G and gc = tc©Pc is the Cartan decomposition of 
the complexification gc of g. We do not know yet how to quantize a nilpotent orbit 
in general. But for a maximal compact subgroup K of G, D. Vogan attaches a space 
with a representation of K to a nilpotent orbit. We explain this correspondence in 
a rough way. Most of the materials in this section come from the article In 
Section 7, we outline the notion of minimal orbits (that are nilpotent orbits), and 
the relation of the minimal representations to the theory of reductive dual pairs 
initiated first by R. Howe. We also discuss the recent works for a construction of 
minimal representations for various groups. For more detail, we refer to |65| . In 
Section 8, we study the orbit method for the Heisenberg group in some detail. In 
Section 9, we study the unitary representations of the Jacobi group and their related 
topics. The Jacobi group appears in the theory of Jacobi forms. That means that 
Jacobi forms are automorphic forms for the Jacobi group. We study the coadjoint 
orbits for the Jacobi group. 

Notation. We denote by Z, R, and C the ring of integers, the field of real num- 
bers, and the field of complex numbers respectively. The symbol C* denotes the 
multiplicative group consisting of all complex numbers z with \z\ = 1, and the sym- 
bol Sp(n, K) the symplectic group of degree n, H n the Siegel upper half plane of 
degree n. The symbol means that the expression on the right hand side is the 
definition of that on the left. We denote by Z + the set of all positive integers, by 
p(k,l) ^jjg ge ^ f a n ^ x Z matrices with entries in a commutative ring F. For any 
M e F (fc ' z) , 'M denotes the transpose matrix of M. For A e F (fc < fc) , a{A) denotes 
the trace of A. For A <E and B E F {k ' k \ we set B[A] = l ABA. We denote the 

identity matrix of degree k by E^. For a positive integer n, Symm(n, K) denotes 
the vector space consisting of all symmetric n x n matrices with entries in a field 
K. 
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2. Quantization 

The problem of quantization in mathematical physics is to attach a quantum 
mechanical model to a classical physical system. The notion of geometric quanti- 
zation had emerged at the end of the 1960s relating to the theory of unitary group 
representations which led to the orbit method. The goal of geometric quantization 
is to construct quantum objects using the geometry of the corresponding classical 
objects as a point of departure. In this paper we are dealing with the group rep- 
resentations and hence the problem of quantization in representation theory is to 
attach a unitary group representation to a symplectic homogeneous space. 

A classical mechanical system can be modelled by the phase space which is a 
symplectic manifold. On the other hand, a quantum mechanical system is modelled 
by a Hilbert space. Each state of the system corresponds to a line in the Hilbcrt 
space. 

Definition 2.1. A pair (M, u>) is called a symplectic manifold with a nondegener- 
ate closed differential 2-form ui. We say that a pair (M, c) is a Poisson manifold if 
M is a smooth manifold with a bivector c = c^didj such that the Poisson brackets 

(2.1) {h,f2} = c^d i f 1 d j f 2 

define a Lie algebra structure on C°°(M). We define a Poisson G- manifold as a 
pair (M, fM) where M is a Poisson manifold with an action of G and fM : g — > 
C°°(M) (A i — ► fM) is a Lie algebra homomorphism such that the following relation 
holds: 

(2.2) s-grad(/^) = L x , A e 0, leg. 

Here Lx is the Lie vector field on M associated with X € g, and s-grad(/) denotes 
the skew gradient of a function /, that is, the vector field on M such that 

(2.3) s-grad(/)<7 = {/,<?} for all g e <7°°(M). 

For a given Lie group G the collection of all Poisson G-manifolds forms the 
category V(G) where a morphism a : (M,fM) — > (N,fP{) is a smooth map from 
M to N which preserves the Poisson brackets: {a*((p), a*(ip)} — a*({<f),ip}) and 
makes the following relation holds: 

(2-4) «*(/£) = /! f , AG fl . 

Observe that the last condition implies that a commutes with the G-action. 

First we explain the mathematical model of classical mechanics in the Hamil- 
tonian formalism. 

Let (M, ui) be a symplectic manifold of dimension In. According to the Darboux 
theorem, the sympletic form u> can always be written in the form 



(2.5) 



u> = Hl =1 dp k A dq k 
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in suitable canonical coordinates pi, ■ ■ ■ ,p n , qi, ■ ■ ■ ,q n . However, these canonical 
coordinates are not uniquely determined. 

The symplectic from lu sets up an isomorphism between the tangent and cotan- 
gent spaces at each point of M. The inverse isomorphism is given by a bivector c, 
which has the form 

(2-6) c = ££ =1 AJL 

opk oqk 

in the same system of coordinates in which the equality (2.5) holds. In the general 
system of coordinates the form ui and the bivecter c are written in the form 

(2.7) u> = Y.i < jLOijdxi A dxj, c = Yii < jC lJ didj 

with mutually inverse skew-symmetric matrices (u>ij ) and (c y ) . The set C°° (M) of 
all smooth functions on M forms a commutative associative algebra with respect 
to the usual multiplication. The Poisson bracket { , } defined by 

(2.8) {F, G} := ZijcVdjF ■ d l G, F,G £ C°°(M) 

defines a Lie algebra structure on C°°(M). The Jacobi identity for the Poisson 
bracket is equivalent to the condition dio = and also to the vanishing of the 
Schouten bracket 

(2.9) [c,c] ijk := ^ ijk H m e m d m c? k 

where the sign -r^ijk denotes the sum over the cyclic permutations of the indices 
i,j,k. 

Physical quantities or observables are identified with the smooth functions on 
M. A state of the system is a linear functional on C°°{M) which takes non-negative 
values on non-negative functions and equals 1 on the function which is identically 
equal to 1. The general form of such a functional is a probability measure /i on M. 
By a pure state is meant an extremal point of the set of states. 

The dynamics of a system is determined by the choice of a Hamiltonian function 
or energy, whose role can be played by an arbitrary function H £ C°°(M). The 
dynamics of the system is described as follows. The states do not depend on time, 
and the physical quantities are functions of the point of the phase space and of time. 
If F is any function on MxR, that is, any observable, the equations of the motion 
have the form 

(2.10) F:=^ = {H,F}. 

Here the dot denotes the derivative with respect to time. In particular, applying 
(2.10) to the canonical variables pk, qk, we obtain Hamilton's equations 
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A set {Fi,--- , F m } of physical quantities is called complete if the conditions 
{Fi, G} = (1 < i < m) imply that G is a constant. 

Definition 2.2. Let (M, oj) be a symplectic manifold and / £ C°°(M) a smooth 
function on M . The Hamiltonian vector field £/ of / is defined by 

(2-12) £/(<?) = {/,<?}, 5 eG°°(M) 

where { , } is the Poisson bracket on C°°(M) defined by (2.8). Suppose G is a Lie 
group with a smooth action of G on M by symplectomorphisms. We say that M is 
a Hamiltonian G- space if there exist a linear map 

(2.13) ft : q — > C°° (M) 

with the following properties (H1)-(H3) : 

(HI) ft intertwines the adjoint action of G on g with its action on G°°(M) ; 
(H2) For each Y e g, the vector field by which Y acts on M is £p,ry); 
(H3) /i is a Lie algebra homomorphism. 

The above definition can be formulated in the category of Poisson manifolds, or 
even of possibly singular Poisson algebraic varieties. The definition is due to A. A. 
Kirillov gHj and B. Kostant |57| . 

The natural quantum analogue of a Hamiltonian G-space is simply a unitary 
representation of G. 

Definition 2.3. Suppose G is a Lie group. A unitary representation of G is a pair 
(it, 7i) with a Hilbert space Tt, and 

7r : G — > U{H) 

a homomorphism from G to the group of unitary operations on TL. 



We would like to have a notion of quantization passing from Definition 2.2 to 
Definition 2.3 : that is, from Hamiltonian G-space to unitary representations. 

Next we explain the mathematical model of quantum mechanics. In quantum 
mechanics the physical quantities or observables are self-adjoint linear operators on 
some complex Hilbert space TL. They form a linear space on which two bilinear 
operations are defined : 

(2.14) A o B := - (AB + BA) (Jordan multiplication) 



(2.15) [A, B} h := — {AB - BA) (the commutator) 

where h is the Planck's constant. 
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With respect to (2.14) the set of observables forms a commutative but not asso- 
ciative algebra. With respect to (2.15) it forms a Lie algebra. These two operations 
(2.14) and (2.15) are the quantum analogues of the usual multiplication and the 
Poisson bracket in classical mechanics. The phase space in quantum mechanics con- 
sists of the non-negative definite operators A with the property that tr A = 1. The 
pure states are the one-dimensional projection operators on TL. The dynamics of 
the system is defined by the energy operator H. If the states do not depend on time 
but the quantities change, then we obtain the Heisenberg picture. The equations of 
motion are given by 

(2.16) A=[H,A]n, (Heisenberg's equation). 

The integrals of the system are all the operators which commute with H. In par- 
ticular, the energy operator itself does not change with time. 

The other description of the system is the so-called Schrodinger picture. In 
this case, the operators corresponding to physical quantities do not change, but the 
states change. A pure state varies according to the law 

2iri - 

(2.17) * = H^f (Schrodinger equation). 

n 

The eigenfunctions of the Schrodinger operator give the stationary states of the 
system. We call a set of quantum physical quantities A\, ■ ■ ■ , A m complete if any 
operator B which commutes with Ai (1 < i < m) is a multiple of the identity. One 
can show that this condition is equivalent to the irreducibility of the set A\, ■ ■ ■ , A m . 

Finally we describe the quantization problem relating to the orbit method in 
group representation theory. As I said earlier, the problem of geometric quantiza- 
tion is to construct a Hilbert space 7i and a set of operators on 7i which give the 
quantum analogue of this system from the geometry of a symplectic manifold which 
gives the model of classical mechanical system. If the initial classical system had a 
symmetry group G, it is natural to require that the corresponding quantum model 
should also have this symmetry. That means that on the Hilbert space Ti. there 
should be a unitary representation of the group G. 

We are interested in homogeneous symplectic manifolds on which a Lie group G 
acts transitively. If the thesis is true that every quantum system with a symmetry 
group G can be obtained by quantization of a classical system with the same symme- 
try group, then the irreducible representations of the group must be connected with 
homogeneous symplectic G-manifolds. The orbit method in representation theory 
intiated first by A. A. Kirillov early in the 1960s relates the unitary representations 
of a Lie group G to the coadjoint orbits of G. Later L. Auslauder, B. Kostaut, M. 
Duflo, D. Vogan, P. Torasso etc developed the theory of the orbit method to more 
general cases. 

For more details, we refer to gZ]- [32], [23, EZj-EHj and [131-121], |22]- 
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3. The Kirillov Correspondence 

In this section, we review the results of Kirillov on unitary representations of a 
nilponent real Lie group. We refer to 03-|SJ, for more detail. 

Let G be a simply connected real Lie group with its Lie algebra g. Let 
Ada ■ G — ► GL(g) be the adjoint representation of G. That is, for each g £ G, 
Adc(g) is the differential map of I g at the identity e, where I g : G — > G is the 
conjugation by g given by 

I g (x) := gxg^ 1 for x £ G. 

Let g* be the dual space of the vector space g. Let Ad* G : G — ► GL(g*) be 
the contragredient of the adjoint representation Ada- Ad* G is called the coadjoint 
representation of G. For each £ £ g*, we define the alternating bilinear form Bi on 
by 

(3.1) B £ {X,Y)=<[X,Y],£>, X,Yeg. 



Definition 3.1. (1) A Lie subalgebra f) of g is said to be subordinate to £ £ g* if 
t) forms a totally isotropic vector space of g relative to the alternating bilinear form 
B e define by (3.1), i.e., B^xft = 0. 

(2) A Lie subalgebra f) of g subordinate to £ £ g* is called a polarization of g for £ 
if f) is maximal among the totally isotropic vector subspaces of g relative to Be. In 
other words, if P is a vector subspace of g such that f) C P and £^|p x p = 0, then 
we have \) = P. 

(3) Let £ £ g* and let f) be a polarization of g for £. We let ff the simply connected 
closed subgroup of G corresponding to the Lie subalgebra f) . We define the unitary 
character \i,t) of H by 

(3.2) X , !f ,(ex Pff (A))=e 2 -<^>, let), 

where exp ff : f) — > _ff denotes the exponential mapping of f) to _ff . It is known that 
exp ff is surjective. 

Using the Mackey machinary, Dixmier and Kirillov proved the following impor- 
tant theorem. 

Theorem 3.1 fDixmier-Kirillov. |19|.|47 1). A simply connected real nilpotent 
Lie group G is monomial, that is, each irreducible unitary representation of G can 
be unitarily induced by a unitary character of some closed subgroup of G. 

Remark 3.2. More generally, it can be proved that a simply connected real Lie 
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group whose exponential mapping is a diffeomorphism is monomial. 
Now we may state Theorem 3.1 explicitly. 

Theorem 3.3 (Kirillov, [47J). Let G be a simply connected nilpotent real Lie 
group with its Lie algebra g. Assume that there is given an irreducible unitary rep- 
resentation 7r of G. Then there exist an element f € g* and a polarization f) of g 
for £ such that tt = Ina^xtfy, where * s the unitary character of H defined by 
(3.2). 

Theorem 3.4 (Kirillov, [47j). Let G be a simply connected nilpotent real He 
group with its Lie algebra g. If £ € Q*, there exists a polarization \) of g for £ 
such that the monomial representation Ind^XiA * s irreducible and of trace class. If 
£' is an element of g* which belongs to the coadjoint orbit AdQ(G)£ and f)' a po- 
larization of g for £' , then the monomial representations Ind^Xi,^ an d I n( %'Xi' ,fj' 
are unitarily equivalent. Here H and H' are the simply connected closed subgroups 
corresponding to the Lie subalgebras f) and f)' respectively. Conversely, if f) and ()' 
are polarizations of g for f G g* and £' £ g* respectively such that the monomial 
representations Ind^xt.t) an d Indjj'Xt'A' of G are unitarily equivalent, then £ and 
£' belong to the same coadjoint orbit of G in g* . Finally, for each irreducible unitary 
representation r of G, there exists a unique coadjoint orbit Q of G in g* such that 
for any linear from £ G and each polarization f) of g for I, the representations r 
and Ind%xt,h are unitarily equivalent. Any irreducible unitary representation of G 
is strongly trace class. 

Remark 3.5. (a) The bijection of the space g* /G of coadjoint orbit of G in g* 
onto the unitary dual G of G given by Theorem 3.4 is called the Kirillov corre- 
spondence of G. It provides a parametrization of G by means of the coadjoint orbit 
space. 

(b) The above Kirillov's work was generalized immediately to the class known as 
exponential solvable groups, which are characterized as those solvable group G 
whose simply-connected cover G is such that the exponential map exp : g — ► G is 
a diffeomorphism. For exponential solvable groups, the bijection between coadjoint 
orbits and representations holds, and can be realized using induced representations 
by an explicit construction using a polarization just as in the case of a nilpotent 
real Lie group. However, two difficulties arise : Firstly not all polarizations yield 
the same representation, or even an irreducible representation, and secondly not all 
representations are strongly trace class. 

Theorem 3.6 (I.D. Brown). Let G be a connected simply connected nilpotent 
He group with its Lie algebra g. The Kirillov correspondence 

G — ► 0{G) = g*/G 

is a homeomorphism. 

Theorem 3.7. Let G be a connected simply connected nilpotent real Lie group with 
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Lie algebra g. Let f) be a subalgebra of g. Let p : g* — ► \)* be the natural projection. 
Let H be the simply connected subgroup of G with its Lie algebra \) . The following 
(a),(b) and (c) hold. 

(a) Let 7r be an irreducible unitary representation of G corresponding to a coad- 
joint orbit Q C g* of G via the Kirillov correspondence. Then ResjjTT decomposes 
into the direct integral of irreducible representations of H corresponding to a coad- 
joint orbit w(C i)*) of H such that ui C 

(b) Let t be an irreducible unitary representation of H corresponding to a coad- 
joint orbit u; C f)* of H. Then the induced representation Lnd%r decomposes into 
the direct integral of irreducible representations -kq, of G corresponding to coadjoint 
orbits C g* such that D u. 

(c) Letiri andiT2 be the irreducible unitary representations of G corresponding to 
coadjoint orbits Sli and respectively. Then the tensor product n 107! 2 decomposes 
into the direct integral of irreducible representations of G corresponding to coadjoint 
orbits f2 C g* such that Q C ill + f^2- 

Theorem 3.8. Let G be a connected simply connected nilpotent real Lie group with 
its Lie algebra g. Let ir be an irreducible unitary representation of G corresponding 
to an orbit Q C g* . Then the character x-n is a distribution on S(G) and its 
Fourier transform coincides with the canonical measure on fl given by the symplectic 
structure. Here S(G) denotes the Schwarz space of rapidly decreasing functions on 
G. 

A. A. Kirillov gave an explicit formula for the Plancherel measure on G. We 
observe that for a nilpotent Lie group G, we may choose a subspace Q of g* such 
that generic coadjoint orbits intersect Q exactly in one point. We choose a basis 
Xi, ■ ■ ■ , Xi,yi, ■ ■ ■ , y n -i in g so that yi, ■ ■ ■ , y n -h considered as linear functionals on 
g*, are constant on Q. Then x\, ■ ■ ■ ,xi are coordinates on Q and hence on an open 
dense subset of 0(G). For every / g Q with coordinates x\, ■ ■ ■ ,xi,we consider the 
skew-symmetric matrix A = (aij) with entries 

aij =< [yi,Vj],f >, l<i,j <n-l. 

We denote by p(x\, ■ ■ ■ ,xi) the Pfaffian of the matrix A. We note that p(x\, ■ ■ ■ ,xi) 
is a homogeneous polynomial of degree 

Theorem 3.9. Let G be a connected simply connected nilpotent Lie group. Then 
the Plancherel measure on G = 0(G) is concentrated on the set of generic orbits 
and it has the form 

9 = p(xi, • • • , xi)dx\ A • • • A dxi 
in the coordinates x\, ■ ■ ■ ,xi. 

4. Auslander-Kostant's Theorem 

In this section, we present the results obtained by L. Auslander and B. Kostant 
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in [3] together with some complements suggested by I.M. Shchepochkina |5D | -|5T ) . 
Early in the 1970s L. Auslander and B. Kostant described the unitary dual of all 
solvable Lie groups of type I. 

Theorem 4.1. A connected, simply connected solvable Lie group G belongs to 
type I if and only if the orbit space 0(G) — Q* /G is a Ta-space and the canonical 
symplectic form a is exact on each orbit. 

Remark 4.2. (a) Let G be a real Lie group with its Lie algebra g. Let Q,i :— 
Ad*(G)£ be the coadjoint orbit containing I. From now on, we write Ad* instead of 
Ad*j. Then is simply connected if a Lie group G is exponential. We recall that 
a Lie group G is said to be exponential if the exponential mapping exp : g — ► G is 
a diffcomorphism. But if G is solvable, Qi is not necessarily simply connected. 

(b) Let G be a connected, simply connected solvable Lie group and for £ E g*, we 
let Gt be the stabilizer at I. Then for any I e J*, we have 7Ti(f2;) = Gt/G®, where 
G° denotes the identity component of Gi in G. 

Let G be a Lie group with Lie algebra q. A pair (£, X ) is called a rigged mo- 
mentum if £ S g* , and \ is a unitary character of Gt such that d\ e — 2ni£\ Be , where 
d\e denotes the differential of % at the identity element eoiGg. We denote by g* igg 
the set of all rigged momenta. Then G acts on g* gg by 

(4.1) g ■ (I, X ) := (Ad* (g)£, X ° I g -i) = (I ° Ad(g-\ X oI g -x) 

for all g e G and (£,x) G 0*igg- Here I g denotes the inner automorphism of G 
defined by I g (x) = gxg -1 (x £ G). We note that x Ig- 1 is a unitary character of 
(*Ad*{G)i = gGeg^ 1 - We denote by O r ig g (G) the set of all orbits in g* igg under the 
action (4.1). 

Proposition 4.3. Let G be a connected, simply connected solvable Lie group. 
Then the following (a) and (b) hold. 

(a) The G-action commutes with the natural projection 

(4.2) n: Q * ngg ^ B * (t, X )»t. 

(b ) For a solvable Lie group G of type L, the projection 7r is surjective and the 
fiber over a point £ £ q* is a torus of dimension equal to the first Betti number 
bi(£le) ofVt t . 

Now we mention the main theorem obtained by L. Auslander and B. Kostant 
in0. 

Theorem 4.4 (Auslander-Kostant). Let G be a connected, simply connected 
solvable Lie group of type I. Then there is a natural bijection between the unitary 
dual G and the orbit space O r i gg (G). The correspondence between G and O r i gg (G) is 
given as follows. Let (£, X ) € 9*ig g - Then there always exists a complex subalgebra p 
of Be subordinate to £. We let L(G, £,x,p) be the space of complex valued functions 
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4> on G satisfying the following conditions 

(4.3) 4>(hg) = x(hWg), h £ G e 
and 

(4.4) (L x + 2ni < t, X >)<j> = 0, X £ p, 

where Lx is the right invariant complex vector field on G defined by X £ gc- Then 
we have the representation T of G defined by 

(4.5) (T(gi)<j>)(9) ■= <t>{99i), 9,91 GG. 

We can show that under suitable conditions on p including the Pukanszky condition 

(4.6) p^W)) = e + p 1 - c n e 

and the condition 

(4.7) codimcp = -rankBi, 

the representation T is irreducible and its equivalence class depends only on the 

rigged orbit containing (£,x)- Here p : ► p* denotes the natural projection 

°f flc on t° P* dual to the inclusion p °-> gc- We denote by Tq the representation T 
of G obtained from (£,x) G 8*igg and p. The correspondence between O r i gg (G) and 
G is given by 



Definition 4.5. Let H be a closed subgroup of a Lie group G. We say that a 
rigged orbit Q' £ O vlgg (H) lies under a rigged orbit £ O r i gg (G) (or equivalently, 
O lies over fi') if there exist rigged momenta (£, x) G and (£', x') £ such that 
the following conditions are satisfied: 

(4.8) p{t)=H, X = X' onHnG t . 

We define the sum of rigged orbits fii and 0,2 as the set of all (£,x) G r igg(G) for 
which there exist {t%,Xi) G fij, i = 1,2, such that 

(4.9) £ = ^ 1+ 4, x = XiX2 onG^nGfe. 

I. M. Shchepochkina [HOI - HO] proved the following. 

Theorem 4.6. Let G be a connected, simply connected solvable Lie group and H 
a closed subgroup of G. Then 



The Method of Orbits for Real Lie Groups 



211 



(a) The spectrum of Indf^Sw consists of those for which fi lies over Q' , 
where Sqi is an irreducible unitary representation of H corresponding to a rigged 
orbit fl' in t)*i gg by Theorem 4-4- 

(b) The spectrum of Res^T^i consists of those Sq> for which fl' lies under fl. 

(c) The spectrum ofTa 1 ® Th 2 consists of those Tq for which f2 lies in Sli + £l2. 



5. The Obstacle for the Orbit Method 

In this section, we discuss the case where the correspondence between irreducible 
unitary representations and coadjoint orbits breaks down. If G is a compact Lie 
group or a semisimplc Lie group, the correspondence breaks down. 

First we collect some definitions. Let G be a Lie group with Lie algebra q and 
let S(G) be the Schwarz space of rapidly decreasing functions on G. We define the 
Fourier transform T$ for / e S(G) by 

(5.1) Tf{l) = [ /(expX) e 2mX ^dX, A e 0*. 

Then (5.1) is a well-defined function on g*. As usual, we define the Fourier transform 
T x of a distribution \ <= S'{G) by 

(5.2) <T x ,T f >=< X J>, f€S(G). 

For / e S(G) and an irreducible unitary representation T of G, we put 

T( f ) = [ /(expX)T(expX)dX 

Then we can see that for an irreducible unitary representation of a nilpotent Lie 
group G, we obtain the following formula 

(5.3) trT(/)= / F f {\)dn\, 

Jn 

where Q is the coadjoint orbit in g* attached to T under the Kirillov correspondence 
and dn\ is the measure on fl with dimension 2k given by the form ^ -Bo A - • • ABq (k 
factors) with the canonical symplcctic form on ft. 

Definition 5.1. Let G be a Lie group with Lie algebra g. A coadjoint orbit f2 in 
0* is called integral if the two dimensional cohomology class defined by the canonical 
two form Bq belongs to H 2 (Q, Z), namely, the integral of Ba over a two dimensional 
cycle in fi is an integer. 



5.1. Compact Lie Groups 
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Let G be a connected and simply connected Lie group with Lie algebra g. Then 
the G-orbits in g* are simply connected and have Kahler structures (not unique). 
These Kahler manifolds are called flag manifolds because their elements are realized 
in terms of flags. Let T be a maximal abelian subgroup of G. Then X = G/T is 
called the full flag manifold and other flag manifolds are called degenerate ones. 
From the exact sequence 

► TT k (G) — 7T k {X) — 7T fc _l(T) — 7Tfe_l(G) — ■ ■ ■ 

and the fact that ni(G) = 7r 2 (G) under the assumption that G is simply connected, 
we obtain 

(5.4) H 2 (X,Z) =tt 2 (X) =^i(T) =Z dimT . 

Let be a coadjoint orbit in g*. We identify g* with g and g£ with gc via the 
Killing form so that t£ goes to tc and the weight P C t£ corresponds to a lattice in 
ii* C ig* = «g. Then n t* is a finite set which forms a single VF-orbit, where W is 
the Weyl group defined as W = N G (T)/Z G (T). 

Proposition 5.2. Let Q\ be the orbit passing through the point i\ e t*. Then 

(1) the orbit tt\ is integral if and only if X € P. 

(2) dimfl\ is equal to the number of roots non- orthogonal to A. 



Let n be an integral orbit of maximal dimension in g*. Let A e f! and let 1} be 
a positive admissible polarization for A. Here the admissibility for A means that f) 
satisfies the following conditions: 

(Al) f) is invariant under the action of G\, 

(A2) f) + f) is a subalgebra of gc- 
Here G\ denotes the stabilizer of G at A. 

Let x\ be the unitary character of G\ defined by 
(5.5) XA(expA) = e 2 ^W, X e flA, 

where Q\ is the Lie algebra of G\. We note that G\ is connected. Let L\ be the 
hermitian line bundle over Q — G/G\ defined by the unitary character xx 01 G\. 
Then G acts on the space T(L\) of holomorphic sections of L\ as a representation 
of G. A. Borel and A. Weil proved that T(L\) is non-zero and is an irreducible 
unitary representation of G with highest weight A. This is the so-called Borel- Weil 
Theorem. Thereafter this theorem was generalized by R. Bott in the late 1950s as 
follows. 

Theorem 5.3 (R. Bott). Let p be the half sum of positive roots of the root system 
for (flC)tc)- Then the cohomology space H k (X,L\) is non-zero precisely when 



(5.6) 



p — iX = w(fi + p) 
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for some fi £ P + , w € W and k = l(w), the length of w. In this case the represen- 
tation of G in H k (il,L\) is equivalent to 7r_^,. 

We note that the Borel-Weil Theorem strongly suggests relating ir\ to il\ and, 
on the other hand, the Bott's Theorem suggests the correspondence ir\ «-> Q\+ p 
which is a bijection between the unitary dual G of G and the set of all integral 
orbits of maximal dimension. It is known that 

(5.7) dini7r A = vol(f2 A+P ). 

The character formula (5.3) is valid for a compact Lie group G and provides an 
integral representation of the character : 

(5.8) XA ( ex pX) = -i- / e w < x >dnA. 

p( x ) Jn x 

In 1990 N.J. Wildberger [SSj proved the following. 
Theorem 5.4. Let $ : C°°{q)' — ► C°°(G)' be the transform defined by 

(5.9) < $(!/), / >=< v,p- (f o exp) >, us C°°( & )', f G C°°(G)'. 

Then for Ad(G) -invariant distributions the convolution operators on G and g are 
related by the transform above : 

(5.10) $(/i) * G $(z/) = $(/x * B !/). 

The above theorem says that $ straightens the group convolution, turning it 
into the abelian convolution on g. This implies the following geometric fact. 

Corollary 5.5. For any two coadjoint orbits f?i,J72 C 0, we let C\ — expfli and 
C*2 = exp O2 . Then the following holds. 

(5.11) Ci-C7 2 Cexp(n 1 + n 2 ). 

5.2. Semisimple Lie Groups 

The unitary dual G u of a semisimple Lie group G splits into different series, 
namely, the principal series, degenerate series, complimentary series, discrete series 
and so on. These series may be attached to different types of coadjoint orbits. 
The principal scries were defined first for complex semisimple Lie groups and for 
the real semisimple Lie group G which admit the split Cartan subalgebra [) C 5. 
These series are induced from characters of the Borel subgroup B D H = exp f) . 
The degenerate series are obtained by replacing B by a parabolic subgroup P D B. 
All these series are in a perfect correspondence with the family of coadjoint orbits 
which have a non-empty intersection with t). An irreducible unitary representation 
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7r of G is said to be a discrete series if it occurs as a direct summand in the regular 
representation R of G on L 2 (G,dg). According to Harish-Chandra, if G is a real 
semisimple Lie group, Gd ^ if and only if G has a compact Cartan subgroup. 
Here Gd denotes the set of equivalent classes of discrete series of G. There is an 
interesting complimentary series of representations which are not weakly contained 
in the regular representation R of G. These can be obtained from the principal 
series and degenerate series by analytic continuation. 

The principal series are related to the semisimple orbits. On the other hand, 
the nilpotent orbits are related to the so-called unipotent representations if they 
exist. In fact, the hyperbolic orbits are related to the representations obtained 
by the parabolic induction and the elliptic orbits are connected to the representa- 
tions obtained by the cohomological parabolic induction. The notion of unipotent 
representations are not still well defined and hence not understood well. Recently 
J.-S. Huang and J.-S. Li |41| attached unitary representations to spherical nilpo- 
tent orbits for the real orthogonal and symplectic groups. The study of unipotent 
representations is under way. For some results and conjectures on unipotent repre- 
sentations, we refer to [T], [16], gT] and |9U ] -[5T ] . 

6. Nilpotent Orbits and the Kostant-Sekiguchi Correspondence 

In this section, we present some properties of nilpotent orbits for a reductive 
Lie group G and describe the Kostant-Sekiguchi correspondence. We also explain 
the work of D. Vogan that for a maximal compact subgroup K of G, he attaches 
a space with a ii'-action to a nilpotent orbit. Most of the materials in this section 
are based on the article [55]. 

6.1. Jordan Decomposition 

Definition 6.1.1. Let GL(n) be the group of nonsingular real or complex n x n 
matrices. The Cartan involution of GL(n) is the automorphism conjugate transpose 
inverse : 

(6.1) 8(g)= t g- 1 , geGL(n). 

A linear reductive group is a closed subgroup G of some GL(n) preserved by 9 and 
having finitely many connected components. A reductive Lie group is a Lie group 
G endowed with a homomorphism ir : G — ► G onto a linear reductive group G so 
that the kernel of ir is finite. 

Theorem 6.1.2 (Cartan Decomposition). Let G be a reductive Lie group with 
7r : G > G as in Definition 6.1.1. Let 

K = G e = {geG\ 9(g) = g} 

be a maximal compact subgroup of G. We write K — ir^ 1 ^), a compact subgroup of 
G, and use dir to identify the Lie algebras of G and G. Let p be the (—l)-eigenspace 
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of d9 on the Lie algebra g of G. Then the map 

(6.2) Kxp — >G, (k,X) h-> k- expX, keK,Xep 

is a diffeomorphism from Kxp onto G. In particular, K is maximal among the 
compact subgroups of G. 

Suppose G is a reductive Lie group. We define a map 9 : G — ► G by 

(6.3) 9{k ■ expX) = k ■ exp(-X), ke K, X G p. 

Then 9 is an involution, that is, the Cartan involution of G. The group of fixed 
points of 9 is K. 

The following proposition makes us identify the Lie algebra of a reductive Lie 
group with its dual space. 

Proposition 6.1.3. Let G be a reductive Lie group. Identify g with a Lie algebra 
ofnxn matrices (cf. Definition 6.1.1). We define a real valued symmetric bilinear 
form on g by 

(6.4) < X,Y >= Retr(XY), X,Y e g. 

Then the following (a),(b) and (c) hold: 

(a) The form < , > is invariant under Ad(G) and the Cartan involution 9. 

(b) The Cartan decomposition g = i + p is orthogonal with respect to the form 

< , >, where 6 is the Lie algebra of K (the group of fixed points of 9) which is the 
(+l)-eigenspace of 9 := d9 on q and p is the (-l)-eigenspace of 9 on q. The form 

< , > is negative definite on 6 and positive definite on p. And hence the form < , > 
is nondegenerate on g. 

(c) There is a G-equivariant linear isomorphism 

a* - a, Ami a 

characterized by 

(6.5) X(Y)=<X X ,Y>, Yeq. 

Definition 6.1.4. Let G be a reductive Lie group with Lie algebra g consisting 
of n x n matrices. An element X G g is called nilpotent if it is nilpotent as a ma- 
trix. An element X G g is called semisimple if the corresponding complex matrix 
is diagonalizablc. An element X G g is called hyperbolic if it is semisimple and its 
eigenvalues are real. An element X G g is called elliptic if it is semisimple and its 
eigenvalues are purely imaginary. 

Proposition 6.1.5 (Jordan Decomposition). Let G be a reductive Lie group 
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with its Lie algebra g and let G = K ■ expp be the Cartan decomposition of g (see 
Theorem 6.1.2). Then the following (l)-(5) hold: 



(1) Any element leg has a unique decomposition 

X = x h + x e + x n 

characterized by the conditions that Xh is hyperbolic, X e is elliptic, X n is nilpotent 
and Xh, X e , X n commute with each other. 

(2) After replacing X by a conjugate under Ad(G), we may assume that Xh G 
p, X e <E t and that X n = E belongs to a standard sl(2) triple. We recall that a 
triple {H, E, F} C g is called a standard st(2) triple, if they satisfy the following 
conditions 

(6.6) 9(E) = -F, 6(H) = -H, [H,E]=2E, [E,F]=H. 

(3) The Ad(G) orbits of hyperbolic elements in g are in one-to-one correspon- 
dence with the Ad(K) orbits in p. 

(4) The Ad(G) orbits of elliptic elements in g are in one-to-one correspondence 
with the Ad(K) orbits in t. 

(5) The Ad(G) orbits of nilpotent orbits are in one-to-one correspondence with 
the Ad(K) orbits of standard sl(2) triples in g. 



6.2. Nilpotent Orbits 

Let G be a real reductive Lie group with Lie algebra g. Let gc be the complex- 
ification of g. We consider the complex special linear group «SX(2,C) which is the 
complexification of SL(2,R). We define the involution 6 : SL(2,C) — ► SL(2,C) 

by 

(6.7) e (g)= t g-\ g e SL(2,C). 
We denote its differential by the same letter 

(6.8) 6 (Z) = - t Z, Zesl(2,C). 

The complex conjugation (T defining the real form SL(2,R) is just the complex 
conjugation of matrices. 

We say that a triple {H, X, Y} in a real or complex Lie algebra is a standard 
triple if it satisfies the following conditions: 

(6.9) [H,X]=2X, [H,Y] = -2Y, [X,Y]=H. 



We call the element H (resp. X, Y) a neutral (resp. nilpositive, nilnegative) element 
of a standard triple {H, X, Y}. 
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We consider the standard basis {H , E ,F } of sl(2, M) given by 

i,m * . (J o) , * - (g ;) . * - (J J) . 

Then they satisfy 

(6.11) [flo,£«] =2^o, [Ho, Jo] =-2F , [S ,F ]=F 
and 

(6.12) e Q (H ) = -H , 9 Q (E ) = -F , 9 (F Q ) = -E . 

We fix a Cartan decomposition g = 5 + p and let 9 be the corresponding Cartan 
involution. We say that a standard triple {H, X, Y} is a Cayley triple in g if it 
satisfies the conditions: 

(6.13) 9(H) = -H, 9{X) = -Y, 9(Y) = -X. 

According to (6.11) and (6.12), the triple {H , E , F } is a Cayley triple in st(2, K). 
For a real Lie algebra g, we have the following theorems. 

Theorem 6.2.1. Given a Cartan decomposition 9 on q 7 any triple {H, X, Y} in 
g is conjugate under the adjoint group Ad(G) to a Cayley triple {H' , X' , Y 7 } in g. 

Theorem 6.2.2 (Jacobson-Morozov). Let X be a nonzero nilpotent element 
in g. Then there exists a standard triple {H, X, Y} in g such that X is nilpositive. 

Theorem 6.2.3 (Kostant). Any two standard triples {H,X,Y}, {H',X, Y'} in 
g with the same nilpositive element X are conjugate under G x , the centralizer of 
X in the adjoint group ofG. 

Let {H, X, Y} be a Cayley triple in g. We are going to look for a semisimplc 
element in g. For this, we need to introduce an auxiliary standard triple attached 
to a Cayley triple. We put 

(6.14) H' = i{X-Y), X' = \{X + Y + iH), Y' = ^(X + Y - iH). 
Then the triple {H', X', Y'} in gc is a standard triple, called the Cayley transform 

of{#,x,y}. 

Since 

9{H') = H', 9(X') = -X\ 9{Y r ) = —Y', 

we have 

(6.15) H'etc and X',Y'ep c . 

Therefore the subalgebra C < H',X',Y' > of gc spanned by H',X',Y' is stable 
under the action of 9. A standard triple in gc with the property (6.15) is called 



218 



Jac-Hyun Yang 



normal. 

Theorem 6.2.4. Any nonzero nil-potent element X £ pc is the nilpositive element 
of a normal triple ( see Theorem 6.2.2). 

Theorem 6.2.5. Any two normal triples {H, X, Y}, {H',X,Y'} with the same 
nilpositive element X is -conjugate, where denotes the centralizer of X in 
the complexification Kc of a maximal compact subgroup K corresponding to the Lie 
algebra t. 

Theorem 6.2.6. Any two normal triples {H, X, Y}, {H, X',Y'} with the same 
neutral element H are -conjugate. 

Theorem 6.2.7 (Rao). Any two standard triples {H,X,Y}, {H',X',Y'} in q 
with X — Y = X' — Y' are conjugate under G X ~ Y , the centralizer of X — Y in G. 
In fact, X — Y is a semisimple element which we are looking for. 

Let Atripie be the set of all J 4d(G)-conjugacy classes of standard triples in g. 
Let Otf be the set of all nilpotent orbits in q. We define the map 

(6.16) Q : Ample — > := M - {0} 

by 

(6.17) Q([{H,X,Y}}):=O x , Ox ■= Ad(G) ■ X, 

where {{H, X, Y}] denotes the G-conjugacy class of a standard triple {H, X, Y}. Ac- 
cording to Theorem 6.2.2 (Jacobson-Morozov Theorem) and Theorem 6.2.3 (Kostant's 
Theorem), the map is bijective. 

We put 

(6.18) h =(°. j), x =\(\ :;), y° = \(] \ 

It is easy to see that the triple {ho, x , y } in sl(2, C) is a normal triple. The complex 
conjugation cto acts on the triple {ho, xo, yo} as follows: 

(6.19) o- (h ) = -h Q , o-o(xo) = yo, o- (yo) = x . 

We introduce some notations. We denote by Mor(s[(2, C), gc) the set of all 
nonzero Lie algebra homomorphisms from s[(2,C) to Qc- We define 

Mor R (s[(2,C),g c ) = Mor(s[(2,C),0 C ) | is defined over M}, 

Mor e (s[(2, C), flc ) = { <\> G Mor(s[(2, C), flc ) I & ° = ° #o }, 



(6.20) Mor R ' 9 (s[(2,C), flc ) -Mor K (s[(2,C), 0c )nMor e (s[(2,C), 0c ), 
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Mor CT (s[(2, C), C ) - { 4> G Mor(s[(2, C), flc ) I a ° = ° }, 

Mor CT < e (s[(2, C), flc) = Mor ff (s[(2, C), 0C ) n Mor e (s[(2, C), 0C ). 

We observe that Mor R (st(2, C),flc) is naturally isomorphic to Mor(s[(2, R), g), the 
set of all nonzero Lie algebra real homomorphisms from sl(2, M.) to g. 

Proposition 6.2.8. Suppose (p be a nonzero Lie algebra homomorphism from 
st(2,C) to a complex reductive Lie algebra Qc- Write 

H = <P(H Q ), E = 4>(E ), F = ^(F ) {see (6.2.10)). 

Then the following hold. 
(1) 

0c = J^Qcik), 

fcez 

where 

9c(k) = {Xe Sc \ [H,X] = kX], kez. 

(2) If we write 

l = flc(0) and u = ^fl C (fc), 

fe>o 

then q = I + u is a Levi decomposition of a parabolic subalgebra of Qc- 

(3) The centralizer of E is graded by the decomposition in (1). More precisely, 

k>0 

(4) The subalgebra l E = q^' E is equal to gf., the centralizer in Qc of the image of (f>. 
It is a reductive subalgebra of Qc- Consequently the decomposition in (3) is a Levi 
decomposition of g E . 

Parallel results hold if {H, E, F} are replaced by 

h = (p(h ), x = <t>{x ), y = <j>(y )- 



Proposition 6.2.9. Suppose G is a real reductive Lie group, and let </>r be an 
element of Mor a (s[(2,C), Qc). Define E R , H R , F R by 

Hr = <^r(-Ho)) Er = <Pr{E ), F r = R (F O ). 

Then the following hold. 

(1) E R , F R are nilpotent, and Hr is hyperbolic. 

(2) If we define L — G Hu to be the isotrophy group of the adjoint action at Hr 
and U = exp(u(~}Q), then Q = LU is the parabolic subgroup of G associated to Hr. 
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(3) The isotrophy group G E " of the adjoint action at E R is contained in Q, and 
respects the Levi decomposition : 

(4) The subgroup L Ew — Q H > E * i$ equal to G^ K , the centralizer in G of the 
image image of </>r. It is a reductive subgroup of G. The The subgroup U Em is 
simply connected unipotent. 

(5) Suppose that <pm.,0 is an element of Mor a 8 (sl(2, C),Q C ). Then G^- 6 is stable 
under the action of 9, and we may take 9 as a Cartan involution on this reductive 
group. In particular, G Eft and G^"- have a common maximal compact subgroup 

The above proposition provides good information about the action of G on the 
cone A/r of all nilpotent orbits in q. 

The following proposition gives information about the action of Kc on the cone 
Afg of all nilpotent elements in pc- 

Proposition 6.2.10. Suppose G is a real reductive Lie group with Cartan decom- 
position G — K ■ expp. Let <f>g G Mor (s[(2, C), flc)- We define 

hg = <f>e(ho), xg = cj)g{x ), y g = 4>g{y ). 

Then we have the following results. 

(1) Xg and yg are nilpotent elements in pc, hg <E tc is hyperbolic and ihg G tc 
is elliptic. 

(2) The parabolic subalgebra q = [ + u constructed as in Proposition 6.2.8 using 
hg is stable under 9. 

(3) If we define Lk '■= (Kc) he and Uk = exp(uC\ 6c), then Qk = LkUk is the 
parabolic subgroup of Kc associated to hg . 

(4) K^ 8 C Qk and K^ 6 respects the Levi decomposition 

K? = (L%)(U%). 



(5) L X jl = K c e ' xe is equal to Kp , the centralizer in Kc of the image of <j>$. 
It is a reductive algebraic subgroup of Kc- The subgroup U K is simply connected 
unipotent. In particular, the decomposition of (4) is a Levi decomposition of Kfc. 

(6) Let (j>M.,e be an element of Mor' 7 ' (sl(2, C), Qc)- Then K^" 6 is stable under a , 
and we may take a as complex conjugation for a compact real form of this reductive 
algebraic group. In particular, Kfc and K"j?' ,e have a common maximal compact 
subgroup 
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6.3. The Kostant-Sekiguchi Correspondence 

J. Sekiguchi [7U] and B. Kostant (unpublished) established a bijection between 
the set of all nilpotent G-orbits in fl on the one hand and, on the other hand, the 
set of all nilpotent ATc-orbits in pc- The detail is as follows. 

Theorem 6.3.1. Let G be a real reductive Lie group with Cartan involution 9 
and its corresponding maximal compact subgroup K . Let a be the complex conju- 
gation on the complexification gc of g. Then the following sets are in one-to-one 
correspondence. 

(a) G-orbits on the cone Mm of nilpotent elements in g. 

(b) G-conjugacy classes of Lie algebra homomorphisms </>r in Mor° r (sl(2, C), flc). 

(c) K-conjugacy classes of Lie algebra homomorphisms <pm.,e in Mor <T,e (s[(2, C), gc). 

(d) Kic-conjugacy classes of Lie algebra homomorphisms <pg in Mor 9 (st(2, C), gc). 

(e) Kc- orbits on the cone Me of nilpotent elements in pc- 

Here G acts on Mor CT (sl(2, C),gc) via the adjoint action of G in g: 

(6.21) ( 5 -0r)(C) =Ad(g)(MO), S€Gand € Mor ff ( S [(2, C), flc ). 

Similarly K and K c act on Mor CT ^(st(2, C), fl C ) and Mor e (sl(2, C), g c ) like (6.21) 
respectively. The correspondence between (a) and (e) is called the Kostant-Sekiguchi 
correspondence between the G-orbits in A/r and the ATc-orbits in Afg. If (j)R,e is an 
element in Mor CT ' (s[(2, C), flc) as in (c) , then the correspondence is given by 

(6.22) E = 4>m,e(E Q ) ^x = <fa,g(x ). (see (6.18)) 

The proof of the above theorem can be found in j5U], pp. 348-350. 

M. Vergne jHZ] showed that the orbits G-E = Ad(G)E and K c ■ x = Ad{K c )x 
are diffeomorphic as manifolds with AT-action under the assumption that they are 
in the Kostant-Sekiguchi correspondence. Here E and x are given by (6.22). 

Theorem 6.3.2 (M. Vergne). Suppose G — K- exp{p) is a Cartan decomposition 
of a real reductive Lie group G and E S fl, x £ pc are nilpotent elements. Assume 
that the orbits G ■ E and Kc • x correspond under the Kostant-Sekiguchi correspon- 
dence. Then there is a K-equivariant diffeomorphism from G ■ E onto Kc ■ x. 

Remark 6.3.3. The Kostant-Sekiguchi correspondence sends the zero orbit to 
the zero orbit, and the nilpotent orbit through the nilpositive element of a Cayley 
triple in fl to the orbit through the nilpositive element of its Cayley transform. 

Remark 6.3.4. Let G • E and Kc • x be in the Kostant-Sekiguchi correspondence, 
where E 6 A/r C fl and x € Me C pc- Then the following hold. 

(1) Gc • E — Gc ■ x, where Gc denotes the complexification of G. 

(2) dim c {Kc • a;) = jdiniR (G • E) = |dimc (G c ■ x). 
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(3) The centralizers G E , K£ have a common maximal compact subgroup K E,X 
which is the centralizer of the span of E and x in K. 

Remark 6.3.5. Let it be an irreducible, admissible representation of a reductive 
Lie group G. Recently Schmid and Vilonen gave a new geometric description of the 
Kostant-Sekiguchi correspondence (cf. (7j3, Theorem 7.22) and then using this fact 
proved that the associated cycle Ass("7r) of 7r coincides with the wave front cycle 
WF(7r) via the Kostant-Sekiguchi correspondence (cf. [77], Theorem 1.4). 

6.4. The Quantization of the Jf-action 

It is known that a hyperbolic orbit could be quantized by the method of a 
parabolic induction, and on the other hand an elliptic orbit may also be quantized 
by the method of cohomological induction. However, we do not know yet how to 
quantize a nilpotent orbit. But D. Vogan attached a space with a representation of 
if to a nilpotent orbit. 

We first fix a nonzero nilpotent element A„ S g*. Let E be the unique element 
in g given from A„ via (6.5). According to Jacobson-Morosov Theorem or Theorem 
6.3.1, there is a non-zero Lie algebra homomorphism 0r from st(2,R) to g with 
<fai(En) = E. We recall that Eq is given by (6.10). 4>r extends to an element 
(f>R € Mor CT (sI(2, C), fjc). After replacing A„ by a conjugate under G, we may assume 
that cj)]g. = (j>s. t e intertwines 0q and 0, that is, <ftu,e S Mor cr,6 '(s[(2, C), gc). 

We define 

(6.23) x = c/)R,e(xo) £ pc, x = - 

The isomorphism in (c), Proposition 6.1.3 associates to x a linear functional 

(6.24) Xeeph \o(Y)=<x,Y>, Y e 0. 

We note that the element Xg is not uniquely determined by A„, but the orbit 
Kc ■ Ag is determined by G ■ A„. According to Theorem 6.3.2, we get a if-equivariant 
diffeomorphism 

(6.25) G-\ n ^K c -\g. 




Definition 6.4.1. (1) Let g* m be the space of purely imaginary-valued linear 
functionals on g. We fix an element Xi m € g* m . We denote the G-orbit of Aj m by 

(6.26) O im :=G-X im = Ad*(G)-X im . 

We may define an imaginary-valued symplectic form u>i m on the tangent space 
(6-27) T Aim (a„„)=g/g A "", 
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where g Aim is the Lie algebra of the isotropy subgroup G Xim of G at X im . We denote 
by Sp(u>i m ) the group of symplectic real linear transformations of the tangent space 

(6.27) . Then the isotropy action gives a natural homomorphism 

(6.28) j : G Aim — > Sp(u im ). 

On the other hand, we let Mp(u>i m ) be the metaplectic group of Sp{uJi m ). That is, 
we have the following exact sequence 

(6.29) 1 — » {1, e} —i. Mp(w im ) — » 5p(^ m ) — » 0. 

Pulling back (6.29) via (6.28), we have the so-called metaplectic double cover of the 
isotropy group G im : 

(6.30) 1 — ► {1, e} — ► G A "" — ► G A "™ . 
That is, G A "" is defined by 

(6.31) G Aim = {( 5 ,m) e G Aim x M P (co im )\ j(g)=p(m)}. 

A representation \ 01 G Aim is called genuine if x(e) = —7. We say that \ lS ac ^ _ 
missible if it is genuine, and the differential of \ is a multiple of Aj m : namely, if 



(6.32) x(expx) = exp(A im (ir)) ■ I, x e g im . 

If admissible representations exist, we say that Xi m (or the orbit O jm ) is admissible. 
A pair (Ai m ,x) consisting of an element Xi m € g* m and an irreducible admissible 
representation \ of G Aim is called an admissible G- orbit datum. Two such are called 
equivalent if they are conjugate by G. 

We observe that if G Xim has a finite number of connected components, an irre- 
ducible admissible representation of G Aim is unitarizable. The notion of admissible 
G-orbit data was introduced by M. Duflo |25j . 

(2) Suppose Xg <E is a non-zero nilpotent element. Let K^ e be the isotropy 
subgroup of Kc at Xg. Define 2p to be the algebraic character of K^ 9 by which it 
acts on the top exterior power of the cotangent space at Xg to the orbit: 

(6.33) 2p(fc):=drt(Af(fc)| (lc/( A. ) .), k^K Xe . 

The differential of 2p is a one-dimensional representation of i^f , which we denote 
also by 2p. We define p G (tp 8 )* to be the half of 2p. More precisely, 

(6.34) p{Z)= l -tv{ad*{Z)\ {ld ^ y ), Ze« A «. 

A nilpotent admissible Kc- orbit datum at Xg is an irreducible algebraic representa- 
tion (t, VV) of K^ e whose differential is equal to p- I T , where I T denotes the identity 
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map on V T . The nilpotent element Xg £ p c is called admissible if a nilpotent ad- 
missible i^c-orbit datum at Xe exists. Two such data are called equivalent if they 
are conjugate. 

Theorem 6.4.2 (J. Schwarz). Suppose G is a real reductive Lie group, K is a 
maximal compact subgroup, and Kc is its complexification. Then there is a nat- 
ural bijection between equivalent classes of nilpotent admissible G-orbit data and 
equivalent classes of nilpotent admissible Kc-orbit data. 

Suppose A„ £ q* is a non-zero nilpotent element. Let Xe £ p£ be a nilpotent 
element which corresponds under the Kostant-Sekiguchi correspondence. We fix a 
nilpotent admissible i^c-orbit datum (r, V T ) at Xe- We let 

(6.35) V T := K c x x e V T 

be the corresponding algebraic vector bundle over the nilpotent orbit Kc ■ Xe = 
Kc/K^". We note that a complex structure on V T is preserved by K but not pre- 
served by G. We now assume that the boundary of Kc • Xe (that is, Kc ■ Xe — Kc-Xe) 
has a complex codimension at least two. We denote by denote by Xr: (A„ , r) the 
space of algebraic sections of V T . Then Xk(Xu,t) is an algebraic representation of 
Kc- That is, if k\ £ K c and s £ X k (X„,t), then (hi • s)(kX g ) = s((k^ 1 k)X g ). We 
call the representation (Kc, Xk(X u ,t)) of Kc the quantization of the K- action on 
G ■ X n for the admissible orbit datum (r, V T ). 

What this definition amounts to is a desideratum for the quantization of the G- 
action on G ■ Xe- That is, whatever a unitary representation 7Tg(A„, t) we associate 
to these data, we hope that we have 

(6.36) ^-finite part of 7r G (A„, t) = X K (X n , r). 

When G is a complex Lie group, the coadjoint orbit is a complex symplectic man- 
ifold and hence of real dimension 4m. Consequently the codimension condition is 
automatically satisfied in this case. 

Remark 6.4.3. Nilpotent admissible orbit data may or may not exist. When 
they exist, there is a one-dimensional admissible datum (to, V To ). In this case, all 
admissible data are in one-to-one correspondence with irreducible representations 
of the group of connected components of K^" ; the correspondence is obtained by 
tensoring with To- If G is connected and simply connected, then this component 
group is just the fundamental group of the nilpotent orbit K c • Xe- 

7. Minimal Representations 

Let G be a real reductive Lie group. Let ir be an admissible representation of 
G. Let g be the Lie algebra of G. Three closely related invariants WF(tt), AS^tf) 
and Ass(tt) in g* which are called the wave front set of 7r, the asymptotic support 
of the character of it and the associated variety of ir respectively, are attached to 
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a given admissible representation tt. The subsets WF(tt), AS(tt), and Ass(tt) are 
contained in the cone Af* consisting of nilpotent elements in g*. They are all invari- 
ant under the coadjoint action of G. Each of them is a closed subvariety of g*, and 
is the union of finitely many nilpotent orbits. It is known that WF(tt) — AS(tt). 
W. Schmid and K. Vilonen [77] proved that Ass(ir) coincides with WF(ir) via the 
Kostant-Sekuguchi correspondence. The dimensions of all three invariants are the 
same, and is always even. We define the Gelfand-Kirillov dimension of tt by 

(7.1) dim G _K tt := i dim WF(tt) 

If 0c is simple, there exist a unique nonzero nilpotent Gc-orbit O m i n C g c of 
minimal dimension, which is contained in the closure of any nonzero nilpotent Gc- 
orbit. In this case, we have O m i n = Ox a , where Ox a is the Gc-orbit of a nonzero 
highest root vector X a . 

A nilpotent G-orbit O C g* is said to be minimal if 

(7.2) dim R C = dim c O mi „, 

equivalently, O is nonzero and contained in O m i n D g*. An irreducible unitary 
representation 7r of G is called minimal if 

(7.3) dim G _K7r = ^-dim c O mm . 



Remark 7.1. (1) If G is not of type A n , there are at most finity many minimal 
representations. These are the unipotent representations attached to the minimal 
orbit O min . 

(2) In many cases, the minimal representations are isolated in the unitary dual 
G u of G. 

(3) A minimal representation tt is almost always automorphic, namely, ir oc- 
curs in L 2 (r\G) for some lattice T in G. The theory of minimal representation 
is the basis for the construction of large families of other interesting automorphic 
representations. For example, it is known that the end of complementary series of 
Sp(n, 1) and F4.1 are both automorphic. 

Remark 7.2. Let (tt, V) be an irreducible admissible representation of it. We 
denote by V K the space of if-finite vectors for 7r, where K is a maximal compact 
subgroup of G. Then V K is a £/(rjc)- m odule. We fix any vector 0^a; T € V K . Let 
U n {Qc) be the subspace of U(gc) spanned by products of at most n elements of 0c- 
Put 

X n (w) := J7„(flc)a;7r- 

D. Vogan [SSj proved that dim A„(-7r) is asymptotic to -n d as n — * 00. Here c(n) 
and d are positive integers independent of the choice of x w . In fact, d is the Gelfand- 
Kirillov dimension of tt. We may say that d = dimG_^7r is a good measurement of 
the size of tt. 
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Suppose F = K or C. Let G be a connected simple Lie group over F and 
K a maximal compact subgroup of G. Let g = t + p be the corresponding Cartan 
decomposition of g. If G/K is hermitian symmetric, all the minimal representations 
are known to be either holomorphic or antiholmorphic. They can be found in the 
list of unitary highest weight modules given in [27|. D. Vogan jHHl proved the 
existence and unitarity of the minimal representations for a family of split simple 
group including Eg, F 4 and all classical groups except for the i?„-case (n > 4), 
which no minimal representation seems to exist. The construction of the minimal 
representations for G2 was given by M. Duflo [2U in the complex case and by D. 
Vogan [HSj in the real case. D. Kazhdan and G. Savin constructed the spherical 
minimal representation for every simple, split, simply laced group. B. Gross and 
N. Wallach |32| constructed minimal representations of all exceptional groups of 
real rank 4. R. Brylinski and B. Kostant JSj-CU gave a construction of minimal 
representations for any simple real Lie group G under the assumption that G/K is 
not hermitian symmetric and minimal representations exist. 

For a complex group G not of type A ni the Harish-Chandra module of the 
spherical minimal representation can be realized on [/(g)/ J, where U(q) is the 
universal enveloping algebra of g and J C U($) is the Joseph ideal of g |48| . It is 
known that Sp(2n, C) has a non-spherical minimal representation, that is, the odd 
piece of the Weil representation. The following natural question arises. 

Question. Are there non-spherical minimal representations for complex Lie 
groups other than Sp(2n, C) ? 

Recently P. Torasso |86j gave a uniform construction of minimal representations 
for a simple group over any local field of characteristic with split rank > 3. He 
constructs a minimal representation for each set of admissible datum associated to 
the minimal orbit defined over F. 

Let ir m i n be a minimal representation of G. It is known that the annihilator 
of the Harish-Chandra module of 7r m i„ in U(g) is the Joseph ideal. D. Vogan 
proved that the restriction of ir m i n to K is given by 

TTminlK = ©™ = o^(/UO + n/3), 

where j3 is a highest weight for the action of K on p, /xq 1S a fixed highest weight 
depending on 7r m i„ and V(jio +nf3) denotes the highest weight module with highest 
weight [io-\-n[3. Indeed, there are two or one possibilities for /3 depending on whether 
G/K is hermitian symmetric or not. 

Definition 7.3. (1) A reductive dual pair in a reductive Lie group G is a pair 
(A, B) of closed subgroups of G, which are both reductive and are centralizers of 
each other. 

(2) A reductive dual pair (A, B) in G is said to be compact if at least one of A and 
B is compact. 
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Duality Conjecture. Let (A, B) be a reductive dual pair in a reductive Lie 
group G. Let 7r m in be a minimal representation of G. Can you find a Howe type 
correspondence between suitable subsets of the admissible duals of A and B by 
restricting % m i n to A x B ? 

In the 1970s R. Howe jSH] first formulated the duality conjecture for the Weil 
representation (which is a minimal representation) of the symplectic group Sp(n, F) 
over any local field F. He ^U] proved the duality conjecture for Sp(n, F) when F 
is archimedean and J. L. Waldspurger |94) proved the conjecture when F is non- 
archimedean with odd residue characteristic. 

Example 7.4. Let G be the simply connected split real group E s . Then K = 
Spin(16) is a maximal compact subgroup of G. We take A = B = Spin(8). It is 
easy to see that the pair (A, B) is not only a reductive dual pair in Spin(16), but 
also a reductive dual pair in G. Let 7r m i„ be a minimal representation of G. J.-S. 
Li [HI] showed that the restriction of n m in to A x B C G is decomposed as follows: 

TmmUxB = ©7^(71") ■ (lT <X> 7r), 

where 7r runs over all irreducible representations of Spin(8) and m(7r) is the multi- 
plicity with which n (8 7r occurs. It turns out that m(7r) = +00 for all tt. 



Example 7.5. Let G be the simply connected quaternionic -Eg with split rank 4. 
We let A = Spin(8) and B = Spin(A,A). Then the pair (A, B) is a reductive dual 
pair in G. Let 7r m i„ be the minimal representation of G. H. Y. Loke 62 proved 
that the restriction of ir m i n to A x _B is decomposed as follows. 

TTrmnUxB = ©7^(71") ■ (7T ® 7r'), 

where tt runs over all irreducible representations of A and n' is the discrete series 
representation of B which is uniquely determined by n. All the multiplicities are 
finite. But they are unbounded. 

The following interesting problem is proposed by Li. 

Problem 7.6. Let (A,B) be a compact reductive dual pair in G. Describe 
the explicit decomposition of the restriction Ti m in\Ay.B of a minimal representation 

Kmin Of G tO A X B. 

Remark 7.7. In 42 , J. Huang, P. Paudzic and G. Savin dealt with the family 
of dual pairs (A,B), where A is the split exceptional group of type Gi and B is 
compact. 

8. The Heisenberg Group H^' h) 

For any positive integers g and h, we consider the Heisenberg group 

h r'' 1) = { ( A ^, «) I Kf* e R (M) , « e k + A* 'A symmetric } 
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with the multiplication law 

[X,H,k)o (X',p',k') = (X + X',p + p',n + k' + X^' - p^'). 
Here IR^.s) (resp. R^' 1 )) denotes the all h x g (resp. h x h) real matrices. 



The Heisenberg group if] 
via the mapping 



H^' h) 9 {X,p,n) 



is embedded to the symplectic group Sp(g - 
V \ 



/£ s 
A E, 







h A* 
£ 9 

o o 



-*A 



€ Sp(<7 + h, 



This Heisenberg group is a 2-step nilpotent Lie group and is important in the study 
of toroidal compactifications of Siegel moduli spaces. In fact, H^' h ^ is obtained as 
the unipotent radical of the parabolic subgroup of Sp(g + h, R) associated with the 
rational boundary component F g (cf. [21] p. 123 or [HU] p. 21). For the motivation 
of the study of this Heisenberg group we refer to |1()3| - |1()7] and jll()| . We refer to 
[53-1102 for more results on H^' h \ 

In this section, we describe the Schrodinger representations of fll 9 '^ and the 

coadjoint orbits of H^' h \ The results in this section are based on the article |1(J8| 
with some corrections. 

8.1. Schrodinger Representations 

First of all, we observe that H^ 11 ^ is a 2-step nilpotent Lie group. It is easy to 
see that the inverse of an element (A, p, k) 6 Hg 3 '^ is given by 

(A, p, k) _1 = (—A, —/i, — k + A l p — \i *A). 

Now we set 

(8.1) [A, p, k] := (0, p, k) o (A, 0, 0) = (A, p., k - p t \). 

Then H^' h) may be regarded as a group equipped with the following multiplication 



(8.2) 



[X,p,k] o [A ,^o,ko] = [A + X ,p + po,K+ k q + X t p a + po^} 



The inverse of [A, p, k] € Hm is given by 



[A, p, k]~ 



-A, —p, — k + X p + p A]. 



We set 
(8.3) 



K = 



{[o,M,«]efli 



p£ 



{h - 9 \ K = 'kg R (h ' h) 
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Then if is a commutative normal subgroup of H^' h \ Let K be the Pontrajagin 
dual of A, i.e., the commutative group consisting of all unitary characters of A. 
Then K is isomorphic to the additive group R^.s) x Symm (ft, R) via 

(8.4) < a, a >= e 2 ™(AV+*«) ; a = [0, m, k] G A, a = (£, £) e A. 
We put 

(8.5) S= { [A, 0,0] e ff4 fl,fc) AeRMjsRM. 
Then 5 1 acts on if as follows: 

(8.6) ox([0,m,k]) = [0,m,k + AVi + /i*A], [A,0,0]eS. 

It is easy to see that the Heisenberg group ^H^' h \<>j is isomorphic to the semi- 
direct product S k A of S 1 and A whose multiplication is given by 

(A, a) ■ (A , a ) = (A + A , a + a A (a )), A, A e 5, a, a € A. 

On the other hand, 5 acts on A by 

(8.7) a* x (a) = (/t + 2kA,k), [A, 0, 0] e S, a A. 

Then, we have the relation < a\(a), a >~< a, a x (a) > for all a € A and a € A. 
We have three types of S-orbits in K. 

Type I. Let k e Sym(ft,R) be nondegenerate. The 5-orbit of a(/c) := (0, k) G A 
is given by 



A e 



= R^' 9 ^. 



(8.8) Oft - {(2kA,k) g A 

Type II. Let (/t, k) e R (M) x Sym (ft, R) with degenerate h ^ 0. Then 
(8.9) 

Type III. Let y e M.( h < 9 \ The 5-orbit 0§ of a(y) := (y, 0) is given by 
(8.10) 6y = {(y,0)} = a(y)- 



= {A + 2kA, k) A e R (M) } C r(*.a) x {«}. 



We have 



/ 



K 



( 



u 



keSym{h,R) ■ 
\k nondegenerate / 



u( u <M(J 



u 



(A,K)eR (h ' 9) xSym(h,R) 
k^O degenerate 
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as a set. The stabilizer S% of S at a(k) = (0, k) is given by 

(8.11) S k = {0}. 

And the stabilizer Sy of S at a(y) = (y, 0) is given by 

(8.12) S fi = {[A,0,0] Ael (M) }=S^l M . 

From now on, we set G — H^ h ^ for brevity. It is known that if is a closed, 
commutative normal subgroup of G. Since (A, /i, k) = (0, fx, k + fj, *A) o (A, 0, 0) for 
(A, (j., n) g G, the homogeneous space X = K\G can be identified with M.( h ' 9 ^ via 

Kg = K o (\, 0, 0) i — ► A, g — (A, /x, k) g G. 

We observe that G acts on X by 

(8.13) (K ff )- 5o = ^(A + Ao,0,0) = A + A , 

where g = (A, /i, k) g G and <7o = (Ao, /io, K o) S G. 
If g = (A, /j, k) g G, we have 

(8.14) fc 9 = (0, m, k + n *A), s 9 = (A, 0, 0) 

in the Mackey decomposition of g = k g o s g (cf. 67]). Thus if 50 = (Ao, A«0) K o) g G, 
then we have 

(8.15) s 9 °5o = (A, 0,0)o (A ,^o,ko) = (A + A , /i , kq + A %) 
and so 

(8.16) fc SgO30 = (0,/x o ,/to +Mo% + A*/io + Mo 4 A). 

For a real symmetric matrix c = *c g R.C 1 '' 1 ) with c ^ 0, we consider the 
one-dimensional unitary representation a c of if defined by 

(8.17) a c ({0,», K )) = e 2 ™( CK U, (0, M> /s)GJf, 

where 7 denotes the identity mapping. Then the induced representation U{a c ) :— 
Ind^ cr c of G induced from a c is realized in the Hilbert space 7i CTt , = L 2 (X, dg, C) = 
i 2 (ffi^'S),^) as follows. If 5o = (A ,mo,k ) g G and x = Kg e X with 5 = 
(A, /i, k) g G, we have 

(8.18) (U go (<J c )f)(x)=a c (k Sg0ga )(f(xg Q )), feH„ c . 
It follows from (8.16) that 

(8.19) {U go (a c )f)(X) = e 2— { C (-o+mo%+2aVo)} / ( A + Ao ), 
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Here, we identified x = Kg (resp. xg = Kgg ) with A (resp. A + A ). The induced 
representation U (cr c ) is called the Schrodinger representation of G associated with 
<j c . Thus U(o~ c ) is a monomial representation. 

Now, we denote by mathcalH 17 " the Hilbert space consisting of all functions 
: G — > C which satisfy the following conditions: 

(1) <f>(g) is measurable measurable with respect to dg, 

(2) ((0, n, k) o g)) = e 2 ™^<j){g) for all g £ G, 

(3) || 0|| 2 :=j x \H9)\ 2 dg <oo, g = Kg, 

where dg (resp. dg) is a G-invariant measure on G (rcsp.X = K\G). The inner 
product ( , ) on H crc is given by 

(01,02)=/ 01 (g) Mg)dg for 0i, 2 G ft CTc . 

JG 

We observe that the mapping $ c : H ac — * Tif 7 " defined by 

(8.20) (Mf))(9) = e 2nia{c ^ X)} f(X) > feH (Ta ,g = (X,(i,K)eG 

is an isomorphism of Hilbert spaces. The inverse \I/ C : H."*- — > H ac of <I> C is given 
by 

(8.21) (* C (0))(A) = 0((A,O,O)), ^r,Ael^». 
The Schrodinger representation U (a c ) of G on TC 7 " is given by 

(8.22) (U go (<7 C )0) (g) = e 2^{c(«o+Mo %+A Vo-Ao V)} ^ (( Aq; , 0) o 5 ) , 

where .go = (Ao, Ho, K o), g — (A, (j,,k) £ G and S W" e . (8.22) can be expressed as 
follows. 

(8.23) {U go (a- c )<f>) (g) = e 2 ™{ c ("o+«+w> %+M f A+2A V«)} + a, 0,0)). 



Theorem 8.1. Lei c be a positive symmetric half-integral matrix of degree h. Then 
the Schrodinger representation U(a c ) of G is irreducible. 

Proof. The proof can be found in |99| . Theorem 3. 

8.2. The Coadjoint Orbits of Picture 

In this subsection, we find the coadjoint orbits of the Heisenberg group H^'' 1 ^ 
and describe the connection between the coadjoint orbits and the unitary dual of 
H^' h ^ explicitly. 

For brevity, we let G := as before. Let g be the Lie algebra of G and 
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let g* be the dual space of g. We observe that g can be regarded as the subalgebra 
consisting of all (g + h) x (g + h) real matrices of the form 



X(a,f3,j) = 



/0 t f3\ 

a (3 7 

-*a 

\0 / 



, a, (3 G 7 = *7 G 



of the lie algebra sp(g + h,M.) of the symplectic group Sp(g + h,M). An easy com- 
putation yields 

[X(a, /3, 7), X(S, e, 0] = *(0, 0, a'e + e*a - /?* J - <5*/3). 

The dual space g* of g can be identified with the vector space consisting of all 
(g + h) x (g + h) real matrices of the form 



/0 ■/ 



F(a,b,c) 



*- 0\_ 



% 

\b c -a 0J 



a,b G 



so that 

(8.24) < F(a, b, c),X(a, (3, 7) >= a(F(a, b, c) X(a, (3, 7)) = 2a( t aa + %/?) + cr(c 7 ). 

The adjoint representation Ad of G is given by Ada{g)X = gXg^ 1 for g G G and 
leg. For g G G and F G g*, gFg^ 1 is not of the form F(a, 6, c). We denote by 
(^F^- 1 ),, the 

fo * 0\ 


0*00 ~ part 
y* * * oy 

of the matrix gFg^ 1 . Then it is easy to see that the coadjoint representation Ad* G : 
G — ► GL(q*) is given by Ad* G {g)F = {gFg~ 1 )*, where g e G and F G g*. More 
precisely, 



(8.25) 



Adg(g)F(a, 6, c) = F(a + cfi,b- cA, c), 



where 5 = (\,[i,k) G G. Thus the coadjoint orbit Q a j of G at F(a, b, 0) G g* is 
given by 

(8.26) ft a , 6 = ^ d c?( G ) F ( a > & > °) = { F (a, & > a sin g le P oint 
and the coadjoint orbit O c of G at F(0, 0, c) G g* with c 7^ is given by 

(8.27) n c = Ad* G (G) F(0, 0, c) = {F(a, b, c)\a, b G R^} 9i x M^. 
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Therefore the coadjoint orbits of G in g* fall into two classes : 

(I) The single point {il a ^ b \a, b <E R (,l ' 9) } located in the plane e = 0. 

(II) The affine planes {^ c |c = *c £ MS h > h \ c ^ o} parallel to the homogeneous 
plane c = 0. 

In other words, the orbit space 0(G) of coadjoint orbits is parametrized by 

c- axis, c ^ 0, c = <c G R^'^; 
(a, b) - plane w xrfM. 

The single point coadjoint orbits of the type il a .5 are said to be the degenerate orbits 
of G in q*. On the other hand, the flat coadjoint orbits of the type f2 c are said to 
be the non-degenerate orbits of G in g* . Since G is connected and simply connected 
2-step nilpotent Lie group, according to A. Kirillov (cf. 07J or 0Hj p. 249, Theorem 
1), the unitary dual G of G is given by 

(8.28) G = (r (m) x R( ft >ff)) ]J jz g R ( ' Ul) | = *b, z ^ o\ , 

where ]J denotes the disjoint union. The topology of G may be described as follows. 
The topology on {c — axis — (0)} is the usual topology of the Euclidean space and 
the topology on {F(a,b,0)\a,b € R^s)} is the usual Euclidean topology. But a 
sequence on the c-axis which converges to in the usual topology converges to the 
whole Euclidean space R^ 1 ' 9 ) x R^ 1 ' 9 ) in the topology of G. This is just the quotient 
topology on g*/G so that algebraically and topologically G = Q*/G. 

It is well known that each coadjoint orbit is a symplectic manifold. We will 
state this fact in detail. For the present time being, we fix an element F of g* once 
and for all. We consider the alternating R-bilinear form Bj? on g defined by 

(8.29) B F (X,Y) =< F, [X,Y] >=< ad* g (Y)F, X >, X,Y e 

where ad* : g — ► End(g*) denotes the differential of the coadjoint representation 
Ad* G : G — > GL(g*). More precisely, if F = F(a,b,c), X = X{a,f3,j), and Y = 
X(8,e,£), then 

(8.30) B F (X, Y) = cr{c(a t e + e*a - 0*8 - 8*0)}. 
For F £ g* , wc let 

G F = {g c G\Ad* G {g)F - F} 

be the stabilizer of the coadjoint action Ad* of G on g* at F. Since Gp is a closed 
subgroup of G, Gp is a Lie subgroup of G. We denote by gp the Lie subalgebra of 
g corresponding to Gf . Then it is easy to show that 

(8.31) g F = radBp = {X G g\ad* g (X)F = 0}. 

Here radBp denotes the radical of Bp in g. Wc let Bf be the non-degenerate al- 
ternating R-bilincar form on the quotient vector space g/rad Bp induced from Bf. 
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Since we may identify the tangent space of the coadjoint orbit flp = G/Gp with 
q/qf = q/ radii p, we see that the tangent space of flp at F is a symplectic vector 
space with respect to the symplectic form Bf. 

Now we are ready to prove that the coadjoint orbit £If — Ad* G {G)F is a sym- 
plectic manifold. We denote by X the smooth vector field on g* associated to X e g. 
That means that for each £ e g* , we have 

(8.32) X(£)=ad* g (X)l. 
We define the differential 2-form Ba F on Hp by 

(8.33) B air (X,Y) =B nF (ad* B (X)F,ad;(Y)F) :=B F (X,Y), 
where X, Y E g. 

Lemma 8.2. B^ F is non- degenerate. 

Proof. Let X be the smooth vector field on g* associated to X e g such that 
Bn F (X,Y) = for &WY with F e g. Since Bn F (X,Y) = B F (X,Y) = for all 
Y e g, lejf. Thus J*f = 0. Hence Bq f is non-degenerate. 

Lemma 8.3. Bq f is closed. 

Proof. If Xi,X 2 ,andX 3 are three smooth vector fields on g* associated to 
Xi, X 2 , X 3 e g, then 

dB aF {X u X 2 ,%) - X^Bn^i^Xs)) - X 2 (B n AXi, %)) + X 3 (B Qf (Xu %)) 
-Bn F ([X 1 ,X 2 ],X 3 ) + Bn F ([X 1 ,X 3 ],X 2 ) - B Qf ( [X 2 , X 3 ] , X 1 ) 
= -<F,[[X 1 ,X 2 ],X 3 ] + [[X 2 ,X 3 ],X 1 ] + [[X3,X 1 ],X 2 ] > 
= (by the Jacobi identity). 

Therefore Bq f is closed. □ 

In summary (flp,Bn F ) is a symplectic manifold of dimension 2hg or 0. 

In order to describe the irreducible unitary representations of G corresponding 
to the coadjoint orbits under the Kirillov correspondence, we have to determine the 
polarizations of g for the linear forms F e g*. 

Case I. F = F(a, b, 0); the degenerate case. 

According to (8.26), = fl a .b = {F(a, b, 0)} is a single point. It follows from 
(8.30) that Hp(X,Y) = for all 1,7 £ 5. Thus g is the unique polarization of g 
for F. The Kirillov correspondence says that the irreducible unitary representation 
7r 0j (, of G corresponding to the coadjoint orbit Oi (, is given by 

(8.34) ^ a , b (expA>,/3, 7 )) = e 2"<^K/?,7)> = ^(W^ 
That is, 7r ,6 is a one-dimensional degenerate representation of G. 
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Case II. F = F(0, 0, c), ^ c = *c e R( h > fe ) : the non-degenerate case. 

According to (8.27), n F = n c = {F(a, 6, c)|o, 5 C R^}. By (8.30), we see 
that 

(8.35) t = { A(0,/3, 7 )|/? e R {h > 9 \ 7 = * 7 e M (/t,?l) } 

is a polarization of g for F, i.e.,t is a Lie subalgebra of g subordinate to F 6 g* 
which is maximal among the totally isotropic vector subspaces of g relative to the 
alternating R-bilinear form Bj?. Let if be the simply connected Lie subgroup of G 
corresponding to the Lie subalgebra { of g. We let 

Xc,t : # — > 
be the unitary character of K defined by 

(8.36) Xct(expX(0,/3,7)) = e 27Ti<F ' x( - o ' ^> = e 2ni < c ^. 

The Kirillov correspondence says that the irreducible unitary representation ir c ^ of 
G corresponding to the coadjoint orbit Qp = Q c is given by 

(8.37) 7r c ,( = Ind^Xc,t- 

According to Kirillov's Theorem (cf. we know that the induced representation 

7r Cj t is, up to equivalence, independent of the choice of a polarization of g for F. Thus 
we denote the equivalence class of 7r Cj t by tt c . ir c is realized on the representation 
space L 2 (R( h ' 9 \d£) as follows: 

(8.38) M<?)/)(0 - ^Mt+SWMfte + A), 
where g = (A, fj,,K)€G and £ e R( h >s). Using the fact that 

ex P A(a,/3, 7 ) = (a, A 7 + - (a*0 - pa)), 

we see that tt c is nothing but the Schrodinger representation U(a c ) of G induced 
from the one-dimensional unitary representation a c of K given by cr c ((0, /i, k)) — 
^Iixio(ck) j -y^ e n0 ^ e that 7r c is the non-degenerate representation of G with central 
character X c ■ Z — ► Cf given by X c ((0, 0, «)) = e 2 «^(^). Here Z = {(0,0,k)|k = 
'« S R^' 1 '} denotes the center of G. 

It is well known that the monomial representation (tt c , L 2 (M.( h ' 9 \ d£)) of G ex- 
tends to an operator of operator of trace class 

(8.39) tt c (0) : L 2 (R (h > 9 \dO — ► L 2 (R^),c^) 

for all (j> € C£°(G). Here C^°(G) is the vector space of all smooth functions on 
G with compact support. We let C^°(g) and C(g*) the vector space of all smooth 
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functions on g with compact support and the vector space of all continuous functions 
on g* respectively. If / S C^°(g), we define the Fourier cotransform 

CF S : G c °°(fl) — C(q*) 

by 

(8.40) (CF s (f)) (F') := / f(X) e 2 ^< F '> x >dX, 

where F' € 0* and dA denotes the usual Lebesgue measure on g. According to A. 
Kirillov (cf. @7|), there exists a measure (3 on the coadjoint orbit Q c w RC*^) x RC 1 ' 9 ) 
which is invariant under the coadjoint action og G such that 

(8.41) tT7r*(0)=/ C^(0oexp)(^')W) 

holds for all test functions € C£°(G), where exp denotes the exponentional map- 
ping of g onto G. We recall that 

<{<t>){f) = I <Kx)fc(x)f)dx, 

Jg 

where <f) € C C °°(G) and / £ L 2 (R( h ' B \dg). By the Plancherel theorem, the mapping 

S{G/Z) 3<p^ nl(<fi) e TC(L 2 (R^\d£)) 
extends to a unitary isometry 

(8.42) tt c 2 : L 2 (G/Z, Xc ) — » frS(L 2 (R< h -»>, d£)) 

of the representation space L 2 (G/Z),Xc) of Indf \ c onto the complex Hilbert space 
HS(L 2 (R( h >a\d£)) consisting of all Hilbert-Schmidt operators on L 2 (R(^»), df), 
where S(G/z) is the Schwartz space of all infinitely differentiable complex-valued 
functions on G/Z = R^ 1 * 9 ) x R^ 1,9 ) that are rapidly decreasing at infinity and 
TG(L 2 (R ( - ,l ' 9 - ) , d£)) denotes the complex vector space of all continuous C-linear map- 
pings of L 2 (R t - h -' 9 \ d£) into itself which are of trace class. 
In summary, we have the following result. 

Theorem 8.4. For F — F(a, b, 0) € g*, the irreducible unitary representation 7r 0j b 
of G corresponding to the coadjoint orbit flp = Q c under the Kirillov correspondence 
is degenerate representation of G given by 

Tr a , b (e X pX(a,f3^))=e 4 ™( taa ~ tb P\ 

On the other hand, for F = F(0, 0, c) e g* with ^ c = f c e M.( h > h \ the irreducible 
unitary representation (tt c , L 2 (J$S h,9 \ d£)) of G corresponding to the coadjoint orbit 
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Sl c under the Kirillov correspondence is unitary equivalent to the Schrodinger repre- 
sentation U(o- c ), L 2 (R( h > 9 \d£)) and this non-degenerate representation 7r c is square 
integrable module its center Z . For all test functions <p £ C^ (G), the character 
formula 

trir 2 c ((j)) =C(4>,c) f 0(O,O, K )e 2 " <T(cK) d K 

holds for some constant C((f>,c) depending on </> and c, where dn is the Lebesgue 
measure on the Euclidean space R( h ' h \ 

Now we consider the subgroup K of G given by 

K = {(0, 0, k) e G | n e R (h < g) , K = t ne R {h < h) }. 

The Lie algebra t of K is given by (8.35). The dual space 6* of 6 may be identified 
with the space 

{F(0, b,c)\be R {h ' 9 \ c = *c G RfrM}. 

We let Ad* K : K — ► GL{t*) be the coadjoint representation of K on 6*. The 
coadjoint orbit uJb, c of K at F(0, b, c) £ 6* is given by 

(8.43) uj biC = Ad* K (K)F(0,b,c) = {F(0,b,c)}, a single point. 

Since K is a commutative group, [1, 6] = and so the alternating R-bilinear form 
B/ on 6 associated to F := F(0,b,c) identically vanishes on t x fi(cf. (8.29)). 6 
is the unique polarization of 6 for F = F(0, b, c). The Kirillov correspondence says 
that the irreducible unitary representation \b,c of K corresponding to the coadjoint 
orbit wt>,c is given by 

(8.44) X6;C (expX(0, /3, 7 )) = e ^<HOAc),x(o,M> = e 2™(2'6,3+c 7 ) 
or 

(8.45) X6,c((0, fj,, «)) = e 2 ™V b » + ™\ (0, M , «) e K 

For 7^ c = *c <E RS h - h \ we let 7r c be the Schrodinger representation of G given by 
(8.38). We know that the irreducible unitary representation of G corresponding to 
the coadjoint orbit 

O c = Ad* G (G) F(0, 0, c) = {F(a, b,c)\a,b€ 

Let p : g* — ► 6* be the natural projection defined by p(F(a,b,c)) — F(o,b,c). 
Obviously we have 

p(n c ) = {f(o, b,c)\be R^ } = U 6eR(fc „j w 6iC . 

According to Kirillov Theorem (cf. 0H| p. 249, Theoreml), The restriction 7r c |jf of 
7r c to K is the direct integral of all one-dimensional representations Xb,c of K (b £ 
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IRC 1,9 '). Conversely, we let \b,c be the element of K corresponding to the coadjoint 
orbit u)b, c of K. The induced representation Ind^- Xb,c is nothing but the Schrodingcr 
representation tt c . The coadjoint orbit f2 c of G is the only coadjoint orbit such that 
f2 c R p~ 1 (uJb.c) is nonempty. 

9. The Jacobi Group 

In this section, we study the unitary representations of the Jacobi group which 
is a semi-product of a a symplectic group and a Heisenberg group, and their related 
topics. In the subsection 9.1, we present basic ingredients of the Jacobi group and 
the Iwasawa decomposition of the Jacobi group. In the subsection 9.2, we find 
the Lie algebra of the Jacobi group in some detail. In the subsection 9.3, we give 
a definition of Jacobi forms. In the subsection 9.4, we characterize Jacobi forms 
as functions on the Jacobi group satisfying certain conditions. In the subsection 
9.5, we review some results on the unitary representations, in particular, the Weil 
representation of the Jacobi group. Most of the materials here are contained in j82| - 
|84j . In the subsection 9.6, we describe the duality theorem for the Jacobi group. In 
the final subsection, we study the coadjoint orbits for the Jacobi group and relate 
these orbits to the unitary representations of the Jacobi group. 

9.1 The Jacobi Group G 3 

In this section, we give the standard coordinates of the Jacobi group G J and 
an Iwasawa decomposition of G J . 

9.1.1. The Standard Coordinates of the Jacobi Group G J 

Let m and n be two fixed positive integers. Let 

5p(n,R) = {MeM (2n>) | l MJ n M = J n } 
be the symplectic group of degree n, where 

j ( E n \ 

is the symplectic matrix of degree n. We let 

H n = { Z e C (,l < n) \Z= t Z, Im Z > } 

be the Siegel upper half plane of degree n. Then it is easy to see that Sp(n, M) acts 
on H n transitively by 

(9.1) M < Z >= {AZ + B){CZ + D)-\ 

where M = ( r ^ J G Sp(n,R) and Z G H n . 
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We consider the Heisenberg group 

A,/i e M (m ' Il) , k e M (m ' m) , k + /i*A symmetric 



< m) = {(A,M,«) 



endowed with the following multiplication law 

(9.2) (A, /z, k) o (A', (if, k) = (A + A', fj, + //, k + k! + A V - /z *A'). 

We already studied this Heisenberg group in the previous section. 
Now we let 

G'l m := Sp(n,K) k fl"^ 

the semidirect product of the symplectic group Sp(n, R) and the Heisenberg group 
H^'™ 1 ^ endowed with the following multiplication law 

(9.3) (M,(A, ) u,k))-(M',(A', m ',k')) = (MM',(A + A',/i + A* / ,K + /s / + AV-/i t A / )) 

with M,M' e Sp(n,M.),(X,fi,K), {X',h',k') G i4 n,m) and (A, /z) := (A,/z)M'. We 
call G^ nm the Jacobi group of degree (n,m). If there is no confusion about the 
degree (n,m) 7 we write G J briefly instead of G J nm . It is easy to see that G J acts 
on H ntTn := H n x C^ 7 ™'") transitively by 

(9.4) (M, (A, /z, «)) • (Z, W) = (M < Z >, (W + XZ + n){CZ + D)- 1 ), 
where M = ( ^ ^ ) e Sp(n, R), (A, /z, «) e fl*"'" and ( z > w ) e ff n,m- 



Now we define the linear mapping 

q . ( R (m,n) x R (m,n)) x ( R (m,n) x R (m,n)) > R (m,m) 

by 

Q((A, /z), (A', M ')) - AV - /i*A', A, /z, A', zz' e R(™<">. 
Clearly we have 

(9.5) *Q(e^) = -Q(r/,0 for al^.fjG 



(9.6) Q(£M, r?M) = Q(f , rj) for allM e Sp(n, R). 

For a reason of the convenience, we write an element of G J as 
g = [M, (A, n, «)] := (E 2n , (A, zz, «)) • (M, (0, 0, 0)). 



240 



Jac-Hyun Yang 



Then the multiplication becomes 

[M, (£, «)] o [M', (£', «')] = [MM', (£ + S'M-\k + k' + Q(Z, ^M- 1 ))]. 

For brevity, we set G = 5p(n,M). We note that the stabilizer K of G at iE n under 
the symplectic action is given by 



t AB = t BA, l AA + *BB = E n 



and is a maximal compact subgroup of G. We also recall that the Jacobi group G J 
acts on i?„ jTO transitively via (9.4). Then it is easy to see that the stabilizer K J of 
G J at (iE n , 0) under this action is given by 



K J = |[fe, (0,0, k)] 



k e K, fc =* fee r(™> to ) 
^ if x { (0,0, k)| k =* k e R< m ' m ) }^K x Symm 2 (M m ). 
Thus on G J /K J = H ntTn , we have the coordinate 

g ■ (iE n ,0) := (Z, := (X + %Y, XZ + fi), g e G J . 

In fact, if ff = [M, (A, i u, k)] e G J with M = e G, 

z = m < i£„ >= (iA + B)(ic + zrr 1 = x + iy, 

W = {i(AA + /xC) + \B + XD}(iC + D)- 1 
= {X(iA + B)+ n(iC + D)}{iC + D)- 1 
= XZ + fi. 



We set 



dX 



and 



d 
dX 



fdX u 
\dX nl 

( 



dX u 

d>JC nn i 



dW 



d 


d 


Mil ' 


dX ln 


a 


d 


ox nl 





d 

' dW 



'dWu 
K dW ml 

( 



dW ln 
dW mn , 



Similarly we set dY = (dYij), dX = (dX pq ), d[i — (d(j, pq ), ■ ■ ■ etc. By an easy 



o a 


dWn 


dW ml 


d 


d 


dW ln 


dW mn 




■ ■ etc. 
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calculation, we have 



d 



1 



dW 2i 



— y- 



d_ 

d 



d 



d 



dX 



d_ d_ , _d_ d 

dX^dZ + dZ + dW + dW ' 
_d_ _ d_ d_ ._d_ d_ 

dY~ l dz~ l dz + l dw dw 



We set 



and 



P+ 



P- 



1/5 . d \ _ d d 



(ImW)Y- 1 



(± 



d 



dZ dW ' 



2 \dX dY 

Let g be the Lie algebra of G and g c its complexification. Then 



0c 



A B 

C - l A 



£(2n,2n) 



B= l B, C 



L C 



We let j := iJ n with J„ 



(9.7) 



E r , 
-E n 



We define an involution a of G by 



a(g) := JgJ \ g e G. 



The differential map da = Ad (J) of a extends complex linearly to the complexifi- 
cation Qc of g. Ad (J) has 1 and -1 as eigenvalues. The (+l)-eigenspace of Ad (J) 
is given by 



(9.8) 



A -B 
B A 



(£i(2n,2n) 



l A + A = 0, B = l B 



We note that fic is the complexification of the Lie algebra t of a maximal compact 
subgroup K = G(~] SO(2n, M) = U{n) of G. The (-l)-eigenspace of Ad (J) is given 

by 



(9.9) 



Pc = 



A B 
B -A 



^(2n,2n) 



A= A, B 



l B 



We observe that pc is not a Lie algebra. But pc has the following decomposition 

p c = p+®p-, 
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where 
(9.10) 
and 
(9.11) 



P+ 



"{(- 



A iX 
iX -X 



Y -iY 
-iY -Y 



l|~i(2n,2n) 



^(2n,2n) 



x = l x 



Y — Y 



We observe that p + and p_ are abelian subalgebras of jjc- Since Ad (J) [A, Y] 
[Ad (J) A, Ad (J)Y\ for all X,Y e g c , we have 

(9.12) [«c,«c] c e c , [6c, Pc] c p c , [pc,Pc] c fi c - 
Since Ad(fc)A = kXk^ 1 (k G K, A e g c ), we obtain 

(9.13) Ad(fc)p + c p+, Ad(fc)p_ c p_. 



For instance, if k = 

(9.14) Ad(fc) 
where 



A B 

-B A 



G if, then 



A ±iA 
±iA -A 



A' ±iX' 
±iA' -A' 



A = *A, 



A' = (A + iB)A 4 (.4 + iB). 



the mapping K 9 



If we identify p_ with Symm (C n ) and K with U(n) as a subgroup of GL{n, C) via 
'A —B\ 

\ — ► A + iB e U(n), then the action of K on p_ is 
compatible with the natural representation pi 1 ! of GL(n,C) on Symm 2 (C") given 

by 

p [1] (g)X =gA* 5 , g e GL(n,C), X G Symm 2 (C"). 
The Lie algebra g of G has a Cartan decomposition 



(9.15) 
where 



= eep, 



i = 
p = 



— B A 

A B 

B -A 



(2n,2ra) 
(2n,2n) 



A + *A = 0, B= l B 
A = t A, B= l B 



Then 9 := Ad (J) is a Cartan involution because 

-B(W, 6{W)) = —B(X, A) + B(Y, Y) > 
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for all W = X + Y, let, Ye p. Here B denotes the Cartan-Killing form for g. 
Indeed, 

(9.16) B(X,Y) = 2(n+l)a(XY), X,Yeg. 

The vector space p is identified with the tangent space of H n at iE n . The corre- 
spondence 

(9.17) I(* _ A B )~A + iB 

yields an isomorphism of p onto Symm 2 (C"). The Lie algebra g J of the Jacobi group 
G J has a decomposition 

(9.18) g J = t J + p J , 
where 

r 7 = {(x, (o, o,«)| let, « - *« g R( m ' m )} , 
p J = {(y,(P,Q,o) | rep, P.QeR^}. 

Thus the tangent space of the homogeneous space £f n>TO = G J /K J at (!E n ,0) is 
given by 

p J p e (M(™' m) x R( n ' m )) p e c (n ' m) . 

We define a complex structure I J on the tangent space p J of P„ jm at iE n by 

(9-19) I J ((J i).(^.Q))==((i Z x )aQ,-P) 

Identifying x R( m <") with C^ 7 ™'™) via 

(9.20) (P,Q)^iP + Q, P,QeR (m ' n \ 
we may regard the complex structure I J as a real linear map 

(9.21) I J (X + iY,Q + iP) = (-Y + iX, -P + iQ), 

where X + iY 6 Symm 2 (C"), Q + iP e C^™ 1 '™). I J extends complex linearly on the 
complexification p^ = p ®r C of p. pc has a decomposition 

(9.22) pc = p:[©P-, 

where p+ (resp. p^ ) denotes the (+z)-eigenspace (resp. (— i)-eigenspace) of I J . Pre- 
cisely, both p+ and p'L are given by 

P + = {((iX -x)' (P ' zP) ) ^eSymm 2 (C"), PeC^J 
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and 



X -iX 
-iX -X 



(P, -iP) 



I £ Symm 2 (C"), P e C (m ' n) 



With respect to this complex structure I J , we may say that / is holomorphic if and 
only if £/ = for all £ e pi. 



We fix g = [M, (I, w k)} e G J with M = JJgG. Let T g : H n — ► H n be 

the mapping defined by (9.4). We consider the behavior of the differential map dT g 
oiT g at (iE n ,0) 



dT g : T {iEnfi) (H n , m ) — T (ZtW) (H n>m ), (Z, W) := g ■ (iE n ,0). 

Now we let a(t) = (Z(t),£(t)) be a smooth curve in H n ^ m passing through (iE n ,0) 
with a'(0) = (V, iP + Q)e T (iBBi o)(H„, m ). Then 

7(t) : = ff-a(t) - 

= (M < >, (£(t) + AZ(t) + /i)(CZ(i) + -D) -1 ) 



is a curve in -ff„ jm passing through 7(0) = (Z, W) with (A, p) = (A, /z)M. Using the 
relation 



d_ 

dt 



(CZ(t) + Dy 1 = -(iC + D)- 1 CZ'{Q){iC + D)-\ 



t=o 



we have 



d_ 

dt 



Z(g; t) = AZ'(0)(iC + D)- 1 + (iA + B) 



t=o 



(CZ^ + D)- 1 



t=o 



= AZ'(0)(iC + D)- 1 - (iA + B){iC + D)- l CZ'(G)(iC + D)- 1 
= {A*(iC + D) - (iA + B) t C} t (iC + D)- 1 Z'(0)(iC + D)- 1 
= '(iC + D^Z'^iiC + D)- 1 . 
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and 

Z(g;t) = (t'(0) + \Z'(0))(iC + D)- 1 



d_ 

dt 



t=0 



+ (?(0)+zA + / i)^ 
: {iP + Q + \Z'(0)){iC + D)- 1 



- (i\ + jl)(iC + D)- 1 CZ'{Q){iC + D)- 1 
= (iP + Q)(iC + D)- 1 

+ \\\iC + D) - (a + /i)*c} \iC + Dy 1 Z' {{)){iC + Dy 1 
= {iP + Q)(iC + D)- 1 +\ t {iC + D)- 1 Z'(0)(iC + D)- 1 . 

Here we used the fact that (iC + D)^ 1 C is symmetric and the relation 

X = \A + fiC, fl = \B + [iD. 

Therefore we obtain 

Z'(g;0) = t {iC + D)- 1 Z'{Q){iC + D)- 1 , 

Ci^^^Ci^C + D)- 1 +\ t (iC + D)- 1 Z'(0){iC + D)- 1 . 

In summary, we have 
Proposition 9.1. Let g = [M, (A, fi; «)] e G J with M = ( ^ e G and 



Zet (Z,W) = g ■ (iE n ,0). Then the differential map dT g : T( iEn - ) (H nym ) — ► 
T (z,w){ H n,m) is given by 

(9.23) («, to) .— ► («(<?), w e Symm 2 (C"), it) G C (m ' n) 

= '(iC + DJ-yiC + D)- 1 , 
w{jg) = w iiC + + A*(iC + Dy 1 v{iC + D)- 1 . 

9.1.2. An Iwasawa Decomposition of the Jacobi Group G J 

First of all, we give the Iwasawa decomposition of G = Sp(n, R). For a positive 
diagonal matrix H of degree n, we put 



t(H) = 



H 
H' 1 
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and for an upper triangular matrix A with 1 in every diagonal entry and B e R( m > ri ) , 
we write 

'A B 



n(A, B) 



*A- 



We let A be the set of such all t(H) and let N be the set of such all n(A, B) such 
that n(A, B) e G, namely, A t B — B l A. It is clear that A is an abelian subgroup 
of G and N is a nilpotent subgroup of G. Then we have the so-called Iwasawa 
decomposition 



(9.24) G = NAK = KAN. 

Now we define the subgroups A J , TV' 7 and N J of G J by 

A J = <! t(H, A) := [t(H), (A, 0, 0)] t{H) e A, A e 



,n)| 



V ' I » l.ii:,/.) := [/((.I. />').(().//.())] 



and 



N J := | n(4 B; n, k) = [n(A, B), (0, fi, «)] 



(m,n) 



(m,m) 



For t(ff, A), t(H', A') e A J , we have 

t(H, A) o t(fr', A') = t(HH', A + X'H- 1 ). 

Thus A J is the semidirect product of M( m '") and D+, where D+ denotes the subgroup 
of GL(n, K) consisting of positive diagonal matrices of degree n. Furthermore we 
have for t(H, A) e A J and h(A, B; /j,, n) E N J 

h(A, B; /j,, k) o t(H, A) = [n(A, B)t(H), (XA^ 1 , /j, - XA~ 1 B 1 A, - f t*A- lt X)] 

and 

t(H,X)oh(A,B;fi, K ) = [t(H)n(A,B),(X,fiH, k + XH*h)] 

HA HB \ \ tt t \ 

H -it A -i) A x ^ iH > K + XH w 

Therefore we have 

A) o B; /z, k) o A)" 1 
= [t(iJ)n(X)t(i7" 1 ), (A - XHA^H- 1 , fiH + XHA~ 1 B t AH, 

k + XHA *B 'A^H'X-n t A~ 1 H * A)] 
= [n(HAH-\HBH), (A - XHA~ 1 H~ 1 ,[iH + XHA~ 1 B l AH, 

K + XHA*B*A- 1 H t X- ^A^H'X)]. 
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Thus there is a decomposition 

(9.25) G J = N J A J K. 
For je G' 7 , one has 

g = [n(A, B)t(H)n, (A, n, k)], neK 
= h(A,B;ii*,K*)ot(H, X*)ok 

with 

A* = AiJ, [i* = fi + XB l A and k* = k + A + \B \XA). 
Recalling the subgroup K J of G J defined by 

K J = { [k, (0, 0, «)] I k e K, K= t ne R (m ' m) }, 

we also have a decomposition 

(9.26) G J = N J A J K J . 
For g e G' 7 , one has 

g=[n(A,B)t(H)k,(X,fi,K)} 
= n{A, B; jl) o t(H, A) o [k, (0, 0, k)] 

with 

A = XA, p, = /i + XA^B'A, k = K + ( f j, + XA- 1 B t A) t X. 

We call the decomposition (9.25) or (9.26) an Iwasawa decomposition of G J . Finally 
wc note that the decomposition (9.25) or (9.26) may be understood as the product 
of the usual Iwasawa decomposition (9.24) of G with a decomposition 

(9.27) H^ m) =N a A 

of the Heisenberg group H^' m) into the group A = { (A, 0,0) | A e R (m ' n) } which 
normalizes the maximal abelian subgroup 

R (m,n) }_ 

9.2. The Lie Algebra of the Jacobi Group G J 

In this section, we describe the Lie algebra g J of the Jacobi group G J explicitly 
First of all, we observe that g of G may be regarded as a subalgebra of q j by 
identifying q with q x {0} and the Lie algebra 1) of the Heisenberg group H^ ,m ^ 
may be regarded as an ideal of q j by identifying f) with {0} x \). We denote by 
the matrix with entry 1 where the i-th row and the j-th column meet, all other 
entries 0. 
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For 1 < a, b,p < m, 1 < q < n, we set 



We observe that the set 



Ba : — 



T ■ — 



^ Eij + Eji 












°\ 





















-(Eij + Eji) 





V o 














/ o 


Eij + Eji 






















ij Eji 













•■ 


V o 














( Eij — Eji 










Q \ 























Eij 


- Eji 





) 


V 














( 





Eij H~~ Eji 


o\ 


















-{E^ + Eji) 












V o 














/0 







\ 






\{E ab - 


VE ba ) 




















\0 







/ 






/ 





\ 








E pq 
















— E 


1 






\ 





) 








/0 


Eqp\ 










E pq 



























\0 


o J 











jsy, T a , D° ab 



1 < i < j < n, 1 < k < I < n, 1 < a < b < m 



form a basis of t J and the set 



{ 



Ai j , Bij , -DpQ , D rs 



1 < i < j < n, 1 < p,r < m, 1 < q, s < n 



form a basis of p J (cf. (9.15)). We note that 



A, 



Aji) Bij 



Bji-> $ij 
2 _ i\2 



'Sji, Tij 



T, 



no _ i~)0 -pji _ t\i _ n 
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Lemma 9.2. 



We have the following commutation relation : 



[Aij,Bki 
[Aij , Ski 
[Aij , Tki 
[Bij,B k i 
[Bij,Ski 
[Bij,Tki 
[Sij , Ski 
[Sij,Tki 
[Tij,Tki 
\D° A- 

[D° ab ,D° cd 

[B'pq , A^j 
[Dpq, Bij 
[B'pqi Sij 

\B 1 pqi D rs 
\B*pq, Aij 
[ B*pqi Bij 
\B*pq, Sij 
[Dpq , Tij 

[ B*pqi D rs 



= SikSji + SuSjk + SjkSu + SjiSik, 
= SikTji + SuTjk + SjkTu + 5jiT ik , 
= dikAji — SuAjk + SjkAu — SjiAik, 
= bikBji + SuBjk + SjkBu + SjiBik, 
= SikSji + SuSjk + SjkSu + SjiSik, 
= SikBji — 5 a Bj k + SjkBu — SjiB ik , 
= SikAji - SuAjk - SjkAu - SjiAik, 
= -SikSji + SuSjk + SjkSu - SjiSik, 
= —SikTji — SuTjk + SjkTu + SjiTik, 
= —SikSji — SuSjk — SjkSu — SjiSik, 
= [ D ab'Bij] = [D ab , S^] = [D ab , T^] = 
= [D° ab ,D pq ] = [D° ab ,D P q]=0, 



3qiDpj 



S D - 

Uqj JSpn 



— [-^PQi -^ij] — 3qi-Dpj -\- Sqj Dpii 

$qi ^pj ~ ^qj ^pi j 
= 0, [Dpq,D rs ]=2S qs D° pr , 



3qiDpj 



SqjDpi, 



— &qi ^pj &qj ^pi 7 

— $qi &pj " &qj &pi 



$qi ^qj ^pi 7 



= 0. 



Here 1 < a,b,c,d,p,r < m, 1 < i,j,k,l,q,s < n and 8^ denotes the Kroncckcr 
delta symbol. 

Proof. The proof follows from a straightforward calculation. □ 
Corollary 9.3. We have the following relation : 

[r 7 ,r 7 ]cr 7 , [F,p J ]cp J , 

[p,f)]cfj, [f),f)]cf). 



Proof. It follows immediately from Lemma 9.2. 



□ 
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Remark 9.4. We remark that the relation 



does not hold. 
Now we set 



[p J ,P J ] c t J 



7° ■ 

^ab ■ 



Y, 



± . 



pq ■ 



-(D pq ±V=lD pq ), 



-Sij, 



IT,, 



Lemma 9.5. 



We have the following commutation relation : 



[ Z ab-> Z cd 

\Y + Y + 

rpq' rs 
[ Y PQ' Z ij 

\Y p + q , Z tj 

\v+ X + 

I pq' ij 

\Y+ x~ 

[ pq ' ij 

\Y~ Y~ 

*• PQ rs 

[Y pq , Z; L - 

\Y~ X + - 

\Xpq ! X ij 



— [ Z ab> Y p%] — [ Z abi Z ij] - [ Z ab> X ij] — 0) 

— -() -Y + 4- f) -Y + 

— -S -Y + - S -Y + 

= o, 

= 0, 



-S Y~ 

°qi*pj 



dqjYpi i 



r) Y~ 4- ft Y~ 

u qi 1 pj ~ u qj 1 pi i 

S -Y + 4- i) -Y + 

Uqi 1 p j T Uqj -I pi , 
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[ z ij> z ti\ = fokZji - 5tiZ% - 5 jk Z+ + 5 jt Zf k , 
\ z %i z u\ = S ikZ~[ - SuZ~ k + S jk Z u - SjiZ7 k , 
\ z tp X ki\ = s ikX jt - 5 jk X u + 5 a X jk - S 3 iX ik , 

\ Z tV Z kl\ = Sik Z jl - 5il Z jk - &3* Z tl - 5 H Z iki 

[Z. j ,X+] = 5 tk X+ + S U X+ + 5 jk X+ + S 3l X+, 
\ Z ij,X7.] = -SikX^ - 5 a Xj k - 5 lk X a - 5jiX7 k , 

\ X tj' X ki\ = \ x iji x ki\ = °i 

\ x t X m\ = -\{5ikZ+ + 6 a Z+ + 5 3k Z+ + 5 3l Z+) 

+ ^Y^&kZji + 5 a Zj k + S jk Z- + 5jiZr k ). 



Proof. It follows from Lemma 9.2. 
Corollary 9.6. The set 



□ 



Z 



hi 



1 < a < b < m, 1 <i < j < n, 1 < k < I < n j 
form a basis of the complexification of t J and the set 

X tj' Y pq 1 < * < 3 < 1 < P < m > 1 < 1 < n 



form a basis o/p^. And Y+ q 
ofp+ and jxr 



1 < i < j < n , 1 < V < m i 1<9<« \ form 



a basis 



y- 



1 <i< j <n, 1 < p < to, 1 < q < n > form a basis 



of p J _. Both p+ and p'f are all abelian subalgebras of Q^. We have the relation 

[«c. pi] c pi, [«o p-] c p'L 

is a subalgebra of and t)c, the complexification oft), is an ideal of 
Proof. It follows immediately from Lemma 9.5. □ 

9.3. Jacobi Forms 

Let p be a rational representation of GL(n, C) on a finite dimensional complex 
vector space V p . Let M. £ R( m ' m ) be a symmetric half-integral semi-positive definite 



252 



Jae-Hyun Yang 



matrix of degree m. Let C°°(H n mi V„) be the algebra of all C°° functions on H n m 
with values in V p . For / S C°°(H ntm , V p ), we define 

(f\ pM [(M,(\^ K ))])(Z, W) 

(9 28) .^ e -27Tia(M[W+\Z+p,](CZ+D)- 1 C) x e 2Tria(M(\Z t \+2\ t W +(K+p t \))) 

x p(CZ + D)~ 1 f(M < Z >,{W + XZ + p){CZ + D)- 1 ), 

where M = ^ e Sp(n, R), (A, /i, «) G £T< n ' m) and (Z, G tf n , m . 

Definition 9.7. Let p and ./Vf be as above. Let 

H { z n ' m) := {(A,/i, k) e fl"^ m) | A, /x G Z( m <"\ k G Z< m < m ) }. 

A Jacobi form of index .M with respect to p on T„ is a holomorphic function 
/ G C°°(H nim , V p ) satisfying the following conditions (A) and (B): 

(A) f\ pM [n/\ = f for all 7 G 1^ := T n x H^' m) . 

(B) / has a Fourier expansion of the following form : 



f(Z,W)= J2 E <T,R)-e 



2nio-(TZ) _ e 27ria{RW) 



T>0 
h al f- i nt ogr a 1 



with c(T, R) f only if (? tR ff\ > 0. 

If n > 2, the condition (B) is superfluous by Kocher principle ( cf. 1 1 1 ( > j Lemma 
1.6). We denote by Jp.M^n) the vector space of all Jacobi forms of index M. with 
respect to p on T n . Ziegler ( cf. 1 1 1 ( ) j Theorem 1.8 or Theorem 1.1 ) proves that 
the vector space J Pl x(r n ) is finite dimensional. For more results on Jacobi forms 
with n > 1 and m > 1, we refer to [U], UnSIHnZI and [TO]] . 

Definition 9.8. A Jacobi form / G J P: m(F) is sa id to be a cusp{ or cuspidal) 
form if ^ ^ > for any T, R with c(T, i?) ^ 0. A Jacobi form / G J pM (r) 

is said to be singular if it admits a Fourier expansion such that a Fourier coefficient 
c(T, R) vanishes unless det ^ l tR j = 0. 

Example 9.9. Let S G Z( 2fe ' 2fc ) be a symmetric, positive definite unimodular even 
integral matrix and c G lS 2k ' m ^ . We define the theta series 

df c (Z,W) := e^ sxzt ^+ 2 ^ cSX ' w ^, ZeH n ,WeC^ n \ 

Aez< 2fc .™> 

We put M. := | 'cSc. We assume that 2k < g + rank(A'f). Then it is easy to see 
that is a singular form in Jk,M^g) ( CI - |H<>| P- 212). 
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9.4. Characterization of Jacobi Forms as Functions on the Jacobi Group 

G J 

In this section, we lift a Jacobi form / e Jp.M^n) to a smooth function <&/ on 
the Jacobi group G J and characterize the lifted function on G J . 

We recall that for given p and A4, the canonical automorphic factor Jm,p ■ 
G J x H n>m — > GL{V p ) is given by 

J M ,p(9, (Z,W)) - e -^MM[W + XZ + p](CZ+D)^C) 

x e 27ri<T(X(AZ t A+2A t W+ K + AI t A))^^ c ,^ + D yl 

where g = (M, (A, fi; k)) <E G J with M = e G. It is easy to see that the 

automorphic factor Jm.p satisfies the cocycle condition: 

(9.29) JmA9i92,(Z,W)) = J M , P (92, (Z, W)) J M , P (9i,92 ■ (Z,W)) 

for all .gi, 32 € G J and (Z, W) G H n . m . 

Since the space H n ^ m is diffcomorphic to the homogeneous space G J / K J , we 
may lift a function / on H n ^ n with values in V p to a function $/ on G J with values 
in V p in the following way. We define the lifting 

(9.30) tp PiM : HHn,m,V p ) — ^(G" 7 ,y p ), := $/ 

by 

*/(5): = (/UA<[ff])(i^n,0) 

= J M A9AiE n ,0))f{g-(iE n ,0)), 

where g & G J and T{Hn,m, V p ) (resp. !F{G J , V p )) denotes the vector space consisting 
of functions on i? n , m (resp. G J ) with values in V p . 

For brevity, we set V := Y n = Sp(n, Z) and r J = T k H^ n ' m) . We let T T p M be 
the space of all functions / on H niTn with values in V p satisfying the transformation 
formula 

(9.31) f\p <M [j] = f for all 7 eP 7 . 

And we let M (G J ) be the space of functions $ : G J — ► V p on G J with values 
in V p satisfying the following conditions (9.32) and (9.33): 

(9.32) $(73) = $(.g) for all 7 e P 7 and g e G J . 

(9.33) $( 5 r(^)) = c 2 ™( M ' t VW" 1 *(5) 1 V r(fc, k) := [jfc, (0, 0; k)] e K J . 
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Lemma 9.10. The space T v M is isomorphic to the space T p M {G J ) via the lifting 

fp,M- 

Proof. Let / e T v pM - If 7 e T J , g e G J and r(fc, k) = [fc, (0, 0; k)] e £T J , then wc 
have 

$/(70) = (/U,m[70])(^,O) 

= ((/U,A<[7])l^[ff])(i^n,0) 

= (/U,x[5])(^n,0) (since / e T T pM ) 

- $/(<?) 

and 

<f> f (gr(k,K)) = J M , P (9 r ( k , K ),( iE n,0)) f(gr(k,K) ■ (iE n ,0)) 

= J M ,p( r ( k > K ),( iE n,0)) J M ,p(9>( iE n,0)) f(9- (i E n,0)) 

= e^^piky^fig). 

Here we identified k = f) eK with A + iBe t/ ( n )' 

Conversely, if $ G T^ >M {G J ) 1 G J acting on i?„ jTO transitively, we may define a 
function /$ on ff„ im by 

(9.34) U(g-(iE n ,0)) := Jm,^, (i£7„, O))" 1 *^). 

Let 7 e T J and (Z,W) = g ■ (iE n ,0) for some g € G J . Then using the cocycle 
condition (9.29), we have 

(h\p,Mh\)(Z, W) = J MtP (i, (Z, W))U{i ■ (Z, W)) 

= JmAi,9- {i E n,0))U(ig- (iE n ,0)) 

= Jm,p(1>9- (i E n,0))JM,ph9,( iE n,0)r 1 'i'(ig) 

= Jm, p (j, g ■ {i E n, 0)) Jm,p(7, 9 ■ {i E n, 0)) _1 

= J M , P (9,(i E n,0))- 1 ^(g) 
= U(g-(iE n ,0)) = U(Z,W). 

This completes the proof. □ 

Now we have the following two algebraic representations T p ,m and Tp,A4 of G J 
defined by 

(9.35) T pM {g)f := f\ pM \g-\ g E G J , / e T T pM 
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(9.36) t p>M (gW) := #GrV), «'eG J , $ € F T p , M (G J ). 

Then it is easy to see that these two models T p ,m and T p ^ are intertwined by the 
lifting (p p .M- 

Proposition 9.11. The vector space J p .x(r„) is isomorphic to the space 
A p .m(T j ) of smooth functions $ on G' 7 with values in V p satisfying the follow- 
ing conditions: 

(la) $(75) = for all 7 G T J . 

(lb) $(gr(fc, «)) = e^^Vfe)- 1 ^) for all .9 G G J , r(fc, k) G i^ 7 . 

(2) Xy* = ^T$ = 0, l<*,j<n. 

(3) For all M G Sp(n,R), the function -0 : G J — > V p defined by 

^(<?):=p(y-*)$(M<7), geG J 
is bounded in the domain Y > Y Q . Here g-(iE n ,0) = (Z, W) with Z = X+iY, Y > 0. 



Corollary 9.12. ^mC^w) is isomorphic to the subspace A° p j^iT' 1 ) of A Pi m(F ) 
with the condition (3 ) the function 5 1 — ► $((?) is bounded. 

9.5. Unitary Representations of the Jacobi Group G J 

In this section, we review some results of Takase(cf. [E2]-|EI]) on the unitary 
representations of the Jacobi group G J . We follow the notations in the previous 
sections. 

First we observe that G J is not reductive because the center of G J is given by 

= *k G R(" l ' m ' 1 S Sym 2 (R m ). 



[E 2n ,{0,0; K )] G G J 

Let dxik) be a normalized Haar measure on K so that J„ d,K(k) = 1 and dz(n) = 
Y[ dnij a Haar measure on Z. We let dj^j — dpc x <iz be the product measure on 

i<j 

i'T' 7 ^i^xZ. The Haar measure d G .j on G J is normalized so that 

/ f(a)dG'(a)= I ([ f(gh)d K j(h))d G j /K j(g) 

Jg j Jg'/k j \Jk> J 

for all / G G C (G J ). 

From now on, we will fix a real positive definite symmetric matrix S G 
Sym 2 (]R m ) of degree m. For any fixed Z = X + iY G H n , we define a measure 
^ s ,z on C( m <") by 

(9.37) dv s .z{W) = (det 2S) n (det F)~ m W) dUdV, 
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where W = U + iV € <£ {m ^ with U, V e R<- m ^ and 
(9.38) K S (Z,W) = e^™^ 8 ^^. 

Let H s z be the complex Hilbert space consisting of all C-valued holomorphic 
functions ip on C^ m '"^ such that / c(m ,„) \(p{W)\ 2 dv s ^ z < +00. The inner product on 
i?S iZ is given by 



We put 



/ <p(W)i/>(W)dv s ,z(W), <p, 4 € H S ,z. 



Vs — Jqs ( see subsection 9.4), 

where <5 denotes the trivial representation of GL(n, C). Now wc define a unitary 
representation Ss,z of H^'™^ by 

(9.39) (3s,z(fr)¥>) (W) - ^(/i" 1 , (Z, VK))- 1 • v>(W - AZ - /*), 

where /i = (A, /x, k) G H^' m ^ and <p e #s,z- It is easy to see that (Ss,z,^s,z) is 
irreducible and E<j, z (0, 0, k) = e - 2m(j(SK \ 

Let 

A" = \t e <£ {n ' n) T= l T, RcT>o| 

be a connected simply connected open subset of C^- n ' n K Then there exists uniquely 

1 

a holomorphic function det 2 on X such that 

(1) (det' Tj 2 = det T for all T e T € X, 

(2) det 2 " T — (det T)i for all T e X n K ( "' n) . 
For any integer fc e Z, we set 

(9.40) det^ T = (det^ t) * , T e X. 

For any g = (a, h) e G J with a £ G, we define an integral operator Ts t z{g) from 
H s ,z to H s ,a<z> by 

(9.41) (7s,z( 5 V) (W) - ^GT 1 , (Z, VF))-V(VF'), 

where (Z',14 7 ') = g~ x ■ (Z,W) and v? e ff<j,z- And for any fixed Z and Z' in 
we define a unitary mapping 

(9.42) L7f, iZ : #s,z — #s,z< 
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by 

(t/f, z <p) (W) = 7 (Z', Z) m ■ [ k s ((Z', W), (Z, WOrVW ^s,z(W), 



where 



l{Z',Z) = det _ * ^ Z - ■det(ImZ')* -det(ImZ)' 



and 

KS ((Z', W), (Z, W)) = e^siw'-wHz'-z)- 1 ). 
For any g — (a, h) G G J , we define a unitary operator T${g) of Hs,iE n by 

(9.43) T S ( 5 ) = T Sta - 1<iEn> (g) o U?- 1<iEn>>iEn . 
We put, for any <ri, C72 G G, 

(9.44) /W 2 )= : ^(p 1 ^^) . 

Then the function /?(<7i,<72) satisfies the cocycle condition 

/3(c r 2,cr 3 )/3(cri(T2,0-3)~ :L /3(c r l,cr2Cr3)/?(o-i,0'2) _:L = 1 

for all cri,cr2,cr3 G G. Thus (3{o\,02) defines a group extension G k Ci by Ci = 
{2: G C I \z\ = 1 }. Precisely, G K Ci is a topological group with multiplication 

(eri, ei) • (cr 2 ,£2) = (c r i<T 2 ,/3(cri,(T2)eie2) 

for all {ai, ei) G G x Ci (i = 1,2). If we put 

^ .-x , , det J(cr _1 , iE n ) _ 

9-45 e(a = — / " , , a- G G, 

V ; V y |det J^- 1 ,*^)! 

then we have the relation 

/3(ci,o- 2 ) 2 = e(cri) • e(<7i<72) _1 ■ e(o- 2 ), 01, ct 2 G G. 
Therefore we have a closed normal subgroup 



(9.46) 



G 2 = j(a,e) GGx Ci e^e^r)- 1 } 



of G2 k Ci which is a connected two-fold covering group of G. Since G2 acts on the 
Heisenberg group H^' m ^ via the projection p : G2 — > G, we may put 
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Now we define the unitary representation ujg of G 2 by 

(9.47) oj s (g) = e m -T s (a,h), g = {{a,e),h) e G J 2 . 

It is easy to see that (d>s> Hs,iE n ) is irreducible and the restriction of u>s to Gi 
is the m-fold tensor product of the Weil representation, 10$ is called the Weil 
representation of the Jacobi group G J . 

We set 

p:G 2 — ► G, p(cr, e) = a, 

p J : Gi — » G J , p J ((a,e),h) = (a,h), 

q : G J ► G, q(a, h) = a, 

q J :G J^G 2: qJ(( a ,e),h) = (a,e). 



Proposition 9.13. Let xs be the character of Z = Sym (R m ) defined by Xs( K ) — 
e 2ni<j(Sn) ^ K g 2. VFe denote by G 2 (xs) the set of all equivalence classes of irre- 
ducible unitary representations r of G 2 such that t(k) = Xs( K ) 1 f or a ^ k E Z. 
We put tt = tt o q J G G*2 /or any tt E G 2 . The correspondence 

TT I ► 7T (g) los 

is a bisection from G 2 to G 2 (xs)- And tt (8 is square-integrable modulo Z if and 
only if tt is square integrable. 

Proof. See [H3 , Proposition 11.8. □ 

Proposition 9.14. Let m be even. We put tt = tt o o g G J /or any 7r e G. Tften 
i/ie correspondence 

TT I ► TT (g> 

is a bisection of G to G J . And tt <S> ojs * s square integrable modulo A if and only if 
tt is square integrable. 

Proof. See EU. □ 
The above proposition was proved by Satake [ZH] or by Berndt [H] in the case 
m = 1. 

Let (p, V p ) be an irreducible representation of K = U (n) with highest weight 
I = (hth)'" Jri) € Z", ?i > • • • > l n > 0. Then p is extended to a rational 
representation of GL(n,C) which is also denoted by p. The representation space 
V p of p has an hermitian inner product ( , ) such that (p(g)u,v) — (u, p(g*)v) for 
all g 6 GL(n,C), u,v € V^>, where <?* = We let the mapping J : G x i?„ — > 
GL(n, C) be the automorphic factor defined by 

J(a, Z) = CZ + D, (T=(q ^) eG - 
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We define a unitary representation t\ of K by 

(9.48) n(k) = p(J(k,iE n )), keK. 

We set Jp.s = Jg* ( cf. subsection 9.4). According to the definition, we have 

J p ,s(g, (Z, W)) = T) S (g, (Z, W)) p(J(a, Z)) 

for all g = (a, h) £ G J and (Z, W) € H n>m . For any g = (a, h) £ G J and (Z, W) £ 
H nt m, we set 

J p .s(g,(Z,W)) = r, s (g,(Z, W)) p(jfaZ)), 
4 J PtS (g, (Z, W)) = r ls (g, (Z, W)) p{ l J{a, Z)), 
J p ,s(g,(Z,W)y = % iS (g, (Z,W)). 

Then for all g £ G J , (Z, W) £ i?„, m and u, v £ V p , we have 

(J p>s (g, (Z, W))u, v) = (u, J PtS (g, (Z, W))*v) 

We denote by E(p, S) the Hilbert space consisting of Vp-valued measurable functions 
(p on H n ^ m such that 

W\ 2 = f (p(Im Z) <p(Z, W), ip(Z, W)) k s (Z, W) d(Z, W), 

JH n , m 

where 

d(Z, W) = (dct Y)-( m+n+ VdXdYdUdV, Z = X + iY, W = U + iV 

denotes a G'^-invariant volume element on H njn . The induced representation 

Ind^j(p ® xs) is realized on E(p, S) as follows: For any g £ G J and ip. £ E(p, S), 
we have 

(indf^xsXsV) (Z, W) = J p , s (g-\(Z, W))- 1 ^- 1 ■ (Z,W)). 

We recall that xs is the unitary character of A defined by xs( K ) = e 27Tlcr ( SK \ k £ Z. 
Let H(p, S) be the subspace of E(p, S) consisting of tp £ E(p, S) which is holomor- 
phic on H n)m . Then H(p, S) is a closed G J -invariant subspace of E(p, S). Let tt p ' S 
be the restriction of the induced representation Ind^j(p (£> xs) to H(p, S). 
Takase(cf. |83j . Theorem 1.1) proved the following 

Theorem 9.15. Suppose l n > n+ y. Then H(p, S) ^ andi^ pS is an irreducible 
unitary representation of G J which is square integrable modulo Z. The multiplicity 
of pi in tt p ' S \k is equal to one. 
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We put 



K 2 =p- 1 (K) = j(fc,e) GKxCi 



e 2 = det J(k, iE n ) 



The Lie algebra { of -K2 an d its Cartan algebra are given by 



A -B 
B A 



(2n,2n) 



A+ f A = 0, B= l B 



and 



1) 



-C 

c 



e 



(2n,2n) 



C = diag(ci,c 2 ,- • • ,c„) 



Here diag (ci, C&, • • • , c„) denotes the diagonal matrix of degree n. We define Aj € f)£ 
by Aj ( ^ ) := yf-\cj. We put 



AT 




TOj £ — Z, mi > ••• > m„, — rrij £ Z for all i,j 



We take an element A = Y^j=i m j^j ^ M + . Let p be an irreducible representation 
of K with highest weight I — (1%, ■ ■ ■ , l n ) € Z™, where lj = mj — m n (1 < j < n— 1). 
Let p^x\ be the irreducible representation of K 2 defined by 



(9.49) 



p(J(k,iE n )), (k,e)eK 2 



Then ptx] is the irreducible representation of K 2 with highest weight A = 
(mi, • • • ,m„) and A 1 — > p\x] is a bijection from M + to K 2l the unitary dual of 
K 2 . 

The following proposition is a special case of 02], Theorem 7.2. 
Proposition 9.16. We have an irreducible decomposition 

u>s =®\m(\)p[x], 

where A runs over 



A = IjXj H Aj G M + (1/ = min{m, n}), 



A 7 £ Z suc/i t/iai /1 > • • • > ^ > 



and i/ie multiplicity m(A) is given by 

m(A) = [] 

l<i<j<m 



i - i 
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where lj = if j > v. Let Gi,d be the set of all the unitary equivalence classes of 
square integrable irreducible unitary representations 0/G2. The correspondence 

n 1 — ► Harish— Chandra parameter of ir 

is a bijection from Gi^d to A + , where 



mi > • • • > m n , rrii — rrij 7^ for all i,j, i ^ j 



See ESJ, Theorem 10.2.4.1 for the details. 

We take an element A = J2'j=i m j^j e M + ■ Let tt x € G2,d be the representation 
corresponding to the Harish- Chandra parameter 

n 

i=i 

The representation ir x is realized as follows (see jSS], Theorem 6.6): Let (p,V p ) 
be the irreducible representation of K with highest weight I = ■ ■ ■ , l n ), Z, = 
rrii — m n ( 1 < j < n ). Let if A be a complex Hilbert space consisting of the V p - 
valued holomorphic functions tp on H„ such that 

\ip\ 2 = j (p(l m Z) <p(Z), p(Z)) ■ (det Im Z) m ™ < +00, 

where dZ is the usual G2-invariant measure on H n . Then ir x is defined by 

(tt x ( 9 )v) (Z) = J x (g-\ Z)-V(ff _1 < Z >) 
for all g = (<j, e) e G2 and if E H x . Here 

J x (g,Z)=p(J(a,Z))- Ji{g,Z) m ", 

where 

t/ r?\ l{c < z >,cr < iE n >) _ x 1 

Ji(<7,Z) = ^ — --j8(o-,ct x )-e-\detJ(a,Z)\* . 

2 j(Z,iE n ) 

Proposition 9.17. Suppose l n > n+ f . VFe pw£ A = E"=i<X? - f) X j € M+ ■ 
Then tt p ' S is an irreducible unitary representation of G J and we have a unitary 
equivalence 

(tt x o q J ) ® uj s — ► tt p ' S o p J 
via the intertwining operator A Pi s : H x (g) Hs,iE n — ► H(p, S) defined by 

(A P ,s(<P ® ^)) ( Z > W) = ( de£2S)"( detImZ)-^ V (Z)(Ul iEn tP)(W) 
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for all ip G H x and ip <G Hs,iE n ■ 

9.6. Duality Theorem for G J 

In this subsection, we state the duality theorem for the Jacobi group G J . 

Let Eij denote a square matrix of degree 2n with entry 1 where the i-th row 
and the j-th column meet, all other entries being 0. We put 

n 

Hi = En - E n+i , n+i (1 < i < n), f} = ^ CHi. 

i=l 

Then () is a Cartan subalgebra of g. Let ej : \) — > C (1 < j < n) be the linear form 
on f) defined by 

ej(Hi) = 5ij, 

where Sij denotes the Kronecker delta symbol. The roots of g with respect to f) are 
given by 

±2e, (1 < i < n), ±e k ± e ; (1 < k < I < n). 
The set $ + of positive roots is given by 

$+ = {2e, (1 < i < n), e k + ei(l<k<l<n)}. 

Let 

9 a = {X eg | [H, X] = a(H)X for allHef)} 

be the root space corresponding to a root a of g with respect to \). We put n = 
E$+ 0a- We denne 



N J = {[cx P X,(0,m,0)] e G J len), 



where exp : g — > G denotes the exponential mapping from g to G. A subgroup 
N 9 of G J is said to be horosherical if it is conjugate to N J , that is, TV 3 = gN J g~ 1 
for some g E G. A horosphcrical subgroup TV 9 is said to be cuspidal for r J = 
r„ k ^"' m) in G J if (N3 n L J )\A9 is compact. Let L 2 (T J \G J , p) be the complex 
Hilbert space consisting of all r J -invariant V^-valued measurable functions $ on 
G J such that ||$|| < oo, where || || is the norm induced from the norm | | on 
E(p,M) by the lifting from H n ^ m to G J . We denote by Lq(L j \G j ', p) the subspace 
of L 2 (L J \G J , p) consisting of functions ip on G J such that ip e L 2 (T J \G J , p) and 

/ <p(ngo)dn = 

JNsnr J \Ns 

for any cuspidal subgroup N 9 of G J and almost all go G G J . Let i? be the right 
regular representation of G J on Lq(T j \G j , p). 

Now we state the duality theorem for the Jacobi group G J . 
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Duality Theorem. Let p be an irreducible representation of K with highest weight 
I — (h, ■ ■ ■ ,l n ) S Z n , lx > I2 > • • • > l n - Suppose l n > n + \ and let M. be a half 
integrable positive definite symmetric matrix of degree m. Then the multiplicity 
m p,M of ir p M in the right regular representation R of G in Lq(T j \G j , p) is equal 
to the dimension of Jp U j^{T n ), that is, 

m PtM = dime J^mC^u)- 

We may prove the above theorem following the argument of JU| in the case m = 
n = 1. So we omit the detail of the proof. 

9.7. Coadjoint Orbits for the Jacobi Group G J 

We observe that the Jacobi group G J is embedded in Sp(n + m, R) via 



(9.50) 



(M, (A, /*,«)) 



I A B Afy-^AX 
A E m fi k 

COD CPfi-D*). 

\0 E m 



A B 
C D 



S Sp(n,R). The Lie algebra g J of 



where (M, (A,/x,/e)) G G J with M 
G J is given by 

(9.51) 3 J = {(AT, (P, Q, fl))|l6j, P, Q e R {m ' n \ R = t Re R (m ' m) } 
with the bracket 



(9.52) 
where 

and 



[(^(i^Qi.fli)), (A: 2 ,(P 2 ,Q 2) P 2 ))] = (X,(P,Q,P)), 



01 61 



ci 



«i 



X 2 = 



a 2 
c-2 



a 2 



X — X1X2 — X2X1, 
P = P\a 2 + Qic 2 - P 2 ai - Q2C1, 
Q = Pib 2 - Qi f a 2 - P^i + Q 2 *oi, 
P = Pi *Q 2 - Oi 'ffe - P 2 *Qi + Q.2 'Pi. 



Indeed, an element (X, (P, Q, R)) in g J with X 
with the matrix 



a b 



S g may be identified 



.53) 



(a 6 

P Q 

c -*fl 

/ 



p 



% c = *c, P = 'P 
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in sp(n + m, R). 

Let us identify Q n+m :— sp(n + m,R) with its dual Q n+m (see Proposition 6.1.3. 
(6.5)). In fact, there exists a G-equivariant linear isomorphism 

Qn+m > Qn+m, A h-> X\ 

characterized by 

(9.54) \(Y) = tr(X x Y), Y e Q n+m . 

Then the dual (q j )* of g J consists of matrices of the form 



(9.55) 



fx p 



z q 

V*<? r 







V 


-*x 



t y, z = f z, r 



There is a family of coadjoint orbits £1$ which have the minimal dimension 2n, 
depending on a nonsingular m x m real symmetric matrix parameter 5 and are 
defined by the equation 



(9.56) 



6 = 



where X = 



with y = y and z — z in (9.55). Let us denote by f) njm 

the Lie algebra of the Heisenberg group H^' m) . Then the family Q s (5 = % 6 G 
GL(m,K)) have the following properties (f21)-(f22): 

(01) Under the natural projection on f)* the orbit f2« goes to the orbit which 
corresponds to the irreducible unitary representation U (6) of the Heisenberg group 
H^' m \ namely, the Schrodinger representation of H^ 1 '" 1 ^ (cf. (8.19)). 

(S12) Under the projection on g* = 5p(n,R)*, the orbit Qg goes to O s i gn (det((5))- 
In fact, there is an irreducible unitary representation ns ( S — t 5, 6 G GL(m, R)) 
of G J (or its universal cover) with properties 



(9.57) 



Res^ ( „ 



^ = 11(8), 



Res G tt$ 



dgn(dct((5)) ) 



where tt± are some representations of G (or its universal cover) corresponding to 
the minimal orbits Q± C Q*. Indeed, n± are two irreducible components of the 
Weil representation of G and 77,5 is one of the irreducible components of the Weil 
representation of G J (cf. (9.47)). These are special cases of the so-called unipotent 
representations of G . We refer to |Hn|-|ni]j ESI for a more detail on unipotent 
representations of a reductive Lie group. 
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Now we consider the case m = n — 1. If 

fa b afj,— bX\ 

A 1 /i K 

c d cfi — dX 

\0 1 ) 

is an element of the Jacobi group G J , then its inverse is given by 

( d 0-6 -fj\ 

cfj, — dX 1 Xb — fia —k 

— c a X 

\ 1 ) 

We put 



X = 











°\ 









1 






/ 





1 
















, r = 














, z = 




















-1 





1 











-1 



















0/ 




vo 








0/ 




vo 










f° 


1 





N 




/o 








N 




/o 








°\ 














, Q = 














, R = 





























1 




















\o 





-1 






\1 








0/ 




\o 


1 





0/ 



Then according to (9.55), X, Y, Z, P, Q, R form a basis for (fl J )*. By an easy compu- 
tation, we see that the coadjoint orbits fix, toy, Qq, of X, Y, Z, P, Q, R 
respectively are given by 



fad + bc 



led 

\ 







-2ab 


(ad + bc) 
0/ 



(bd-ac a 2 -6 2 0^ 

d 2 - c 2 ac-bd 



ad — be — 1, a, 6, c, € 



ad — bc= 1, a, b,c 7 d E 



flz = < 



I V 



\ o o o o/ 

' f-(ac + bd) a 2 + 6 2 0^ 



(c 2 + d 2 ) ac + M 

0/ 

( (2ad — 1)A — 2acfi a 2abX - 2a 2 fi 



2c 2 n-2cdX c (1 - 2ad)X + 2ac[i 

k \ c -a 0/ 



> • 



ad — be — 1, a, 6, c, <i G 



0\ 




ad — bc= 1, 
a, 6, c, d,X,fiE 



} ■ 
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fir 



and 



fii? = < 



((2ad-l)n-2bd\ b 2b 2 X-2ab^i 0\ 


2cd/x - 2d 2 X d (1 - 2ad)fi + 2bd\ 



V 







ad — bc= 1, 
a, b, c, d, A, (i e 



0/ 



^ (a/z — b\){cfi — dX) ay, — b\ 


(c/j, — d\) 2 c[i — dX 
cfx — dX 1 







I V 

Moreover we put 



-{a^-bX) 2 


-(afi — bX)(cfi — dX) 
bX — a[i Oj 



[0 1 Q\ 





\0 0/ 



and T 

























1 



















v 



Then the coadjoint orbits fig and fir of S and T are given by 



fis = < 



f /-aft a 2 0\ 



-b 2 ab 

\ 0/ 



a,b E 



and 



fiv 








-a 2 
















6 2 





—ab 





v0 








0/ 



a,b E 



> . 



For an element of ($) )*, we write 



(9.58) 



fx p y + z 0\ 



y — z g —a; 

\ q r -p 0/ 



ad — be = 1 , 
a, 6, c, d, A, /i G 



= il + yY + + pP + gQ + ri?. 



The coadjoint orbit fijf is represented by the one-sheeted hyperboloid 
(9.59) 



x 2 + y 2 - z 2 = 1 > 0, p = q = r = 0. 



The coadjoint orbit fly is also represented by the one-sheeted hyperboloid (9.59). 
The coadjoint orbit fiz is represented by the two-sheeted hyperboloids 



(9.60) 



2 2 

y = z 



1 > 0, p = g = r = 0. 
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The coadjoint G -orbit 0,$ of S is represented by the the cone 

(9.61) x 2 + y 2 = z 2 > 0, 2>0, p = q = r = 0. 

On the other hand, the coadjoint G J -orbit Qt of T is represented by the cone 

(9.62) x 2 +y 2 = z 2 >0, z < 0, p = q = r = 0. 
The coadjoint orbit ilp is represented by the variety 

(9.63) 2pqx+(q 2 - P 2 )y+(p 2 +q 2 )z = 0, (p, q) E R 2 - {(0, 0)}, r = 

in R 6 . The coadjoint orbit orbit is represented by the variety (9.63) in R 6 . 
particular, we are interested in the coadjoint orbits orbits &hR (h E R, h ^ 0) of 
which are represented by 

(9.64) x 2 + y 2 = z 2 , .x = h~ 1 pq 1 y + z = —h~ 1 p 2 , y — z = h~ 1 q 2 and r = h. 

For a fixed ft ^ 0, we note that 17/^ is two dimensional and satisfies the equa- 
tion (9.56). Indeed, from the above expression of Qhr and (9.58), we have 

X=( X y + Z )and 
\y-z -x J 

x = h(a/j, — feA)(c/i — dX), 
y + z = —h(afi — b\) 2 , 
y — z = h(cfi — dX) 2 , 

p = h(afi — bX), q — h(c/i — dX), r = h. 

Hence these satisfy the equation (9.56). An irreducible unitary representation 
iTh that corresponds to a coadjoint orbit £lhR satisfies the properties (9.57). In 
fact, TTh is one of the irreducible components of the so-called (Schrodinger-)Weil 
representation of G J (cf. (9.47)). A coadjoint orbit fl m n +a x or fl m R+aY (m € 
M x , a G M) is corresponded to a principal series 7r mQ i, the coadjoint orbit 
&>mR+kz {m G R x , k G Z + ) of mR + kZ is attached to the discrete series 7r TO k 
of G J . There are no coadjoint G -orbits which correspond to the complimentary 
series n m , a ,v (m E R x , a e R, a 2 < |, v — ±|). See [TT].pd. 47-48. There are no 
unitary representations of G J corresponding to the G^-orbits of aP* -I- with 
(a,P) ^(0,0). 

Finally we mention that the coadjoint orbit Q m R+aX or r2 m /j +Q ,y (m E R x , a € 
R) is characterized by the variety 

2 1 1 

(9.65) x 2 + y 2 — (z 2 + a 2 ) — — pqx H (q 2 — p 2 )y H (p 2 + q 2 )z, r = m. 

mm m 

and the coadjoint orbit Vt m R + kz{ r m E R x ,/c E Z + ) of mR + kZ is represented by 
the variety 

2 1 1 

(9.66) x 2 +y 2 - (z 2 - k 2 ) = —pqx H (q 2 - p 2 )y H (p 2 +q 2 )z, z > 0, r = m. 

mm m 
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2 1 1 

(9.67) x 2 + y 2 - (z 2 - k 2 ) = — pqx + — (q 2 - p 2 )y + — (p 2 + q 2 )z, z < 0, r = m 

mm m 

depending on the sign ±. 
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